Max_len = min(batch_max_len, self.max_len) # compute length of longest sentence in batchīatch_max_len = max() # make a list that decides the order in which we go over the data- this avoids explicit shuffling of dataįor i in range(data//self.batch_size): Shuffle: (bool) whether the data should be shuffledīatch_data: (tensor) shape: (batch_size, max_len)īatch_tags: (tensor) shape: (batch_size, max_len)
Tokenize pandas column generator#
"""Returns a generator that yields batches data with tags.ĭata: (dict) contains data which has keys 'data', 'tags' and 'size' Raise ValueError("data type not in ")ĭef data_iterator(self, data, shuffle=False): Self.load_sentences_tags(sentences_file, tags_path, data) Tags_path = os.path.join(self.data_dir, data_type, 'tags.txt')
Sentences_file = os.path.join(self.data_dir, data_type, 'sentences.txt') """Loads the data for each type in types from data_dir.ĭata_type: (str) has one of 'train', 'val', 'test' depending on which data is required.ĭata: (dict) contains the data with tags for each type in types. # checks to ensure there is a tag for each token Maps tokens and tags to their indices and stores them in the provided dict d. """Loads sentences and tags from their corresponding files. Self.tokenizer = om_pretrained(bert_model_dir, do_lower_case=True)įile_path = os.path.join(self.data_dir, 'tags.txt')ĭef load_sentences_tags(self, sentences_file, tags_file, d): Stop_words = Ĭlean_sentence=' '.join()ĭata = You can try with the following: import pandas as pd
0 kommentar(er)
