3

I have a pandas dataframe A with column keywords as :-

 keywords
 ['loans','mercedez','bugatti','a4']
 ['trump','usa','election','president']
 ['galaxy','7s','canon','macbook']
 ['beiber','spiderman','marvels','ironmen']
 .........................................
 .........................................
 .........................................

I also have another pandas dataframe B with column category and words which is comma seperated string as:-

category              words
audi                  audi a4,audi a6
bugatti               bugatti veyron, bugatti chiron
mercedez              mercedez s-class, mercedez e-class
dslr                  canon, nikon
apple                 iphone 7s,iphone 6s,iphone 5
finance               sales,loans,sales price
politics              donald trump, election, votes
entertainment         spiderman,captain america, ironmen
music                 justin beiber, rihana,drake
........              ..............
.........             .........

All I want to map dataframe A column keywords with dataframe B column words and assign a corresponding category. Mapping of keywords column should be with each word in string of column word. For example:- keyword a4 should be matched with both words in string audi a4 in column words.Expected result would be:-

  keywords                                       matched_category
  ['loans','mercedez','bugatti','a4']            ['finance','mercedez','mercedez','bugatti','bugatti','audi']                                    
  ['trump','usa','election','president']         ['politics','politics']                                           
  ['galaxy','7s','canon','macbook']              ['apple','dslr']
  ['beiber','spiderman','marvels','ironmen']     ['music','entertaiment','entertainment','entertainment']
Learner
  • 800
  • 1
  • 8
  • 23
  • From what I can see most of your words and keywords have some overlaps. You should be able to work with that. – Kwright02 Aug 15 '18 at 19:02
  • @Kwright02 After mapping the keywords, I want to remove the duplicates too. – Learner Aug 15 '18 at 19:05
  • Use a 2d array to iterate over the set of words and if at any point list[i].equals(list[j]) then delete one of them, but make sure that J isn't the same index as I. – Kwright02 Aug 16 '18 at 02:31

2 Answers2

0

One way is to use pandas.transform:

import pandas as pd

A = pd.DataFrame({'keywords': [['loans','mercedez','bugatti','a4'],
                           ['trump','usa','election','president']]})
B = pd.DataFrame({'category': ['audi', 'finance'],
                  'words': ['audi a4,audi a6', 'sales,loans,sales price']})

def match_category_to_keywords(kws):
    ret = []
    for kw in kws:
        m = B['words'].transform(lambda words: any([kw in w for w in words.split(',')]))
        ret.extend(B['category'].loc[m].tolist())
    return pd.np.unique(ret)

A['matched_category'] = A['keywords'].transform(lambda kws: match_category_to_keywords(kws))
print(A)

Output:

                            keywords matched_category
0     [loans, mercedez, bugatti, a4]  [audi, finance]
1  [trump, usa, election, president]               []
Ghasem Naddaf
  • 862
  • 5
  • 15
  • This is totally wrong then the expected output. How you can add multiple categories for dataframe `B`? – James Aug 16 '18 at 05:15
  • Each entry in the list represents one row in B. I only added 2 rows of your data in above example. If you add all rows, you'd get the expected output. – Ghasem Naddaf Aug 16 '18 at 05:18
0

I hope you can use:

#create dictionary by split comma and whitespaces
d = df2.set_index('category')['words'].str.split(',\s*|\s+').to_dict()
#flatten lists to dictionary
d1 = {k: oldk for oldk, oldv in d.items() for k in oldv}
print (d1)
{'audi': 'audi', 'a4': 'audi', 'a6': 'audi', 'bugatti': 'bugatti', 
 'veyron': 'bugatti', 'chiron': 'bugatti', 'mercedez': 'mercedez', 
 's-class': 'mercedez', 'e-class': 'mercedez', 'canon': 'dslr', 
 'nikon': 'dslr', 'iphone': 'apple', '7s': 'apple', '6s': 'apple',
 '5': 'apple', 'sales': 'finance', 'loans': 'finance', 'price': 'finance', 
 'donald': 'politics', 'trump': 'politics', 'election': 'politics', 
 'votes': 'politics', 'spiderman': 'entertainment', 'captain': 'entertainment',
 'america': 'entertainment', 'ironmen': 'entertainment', 'justin': 'music', 
 'beiber': 'music', 'rihana': 'music', 'drake': 'music'}

#for each value map in nested list comprehension
df1['new'] = [[d1.get(y, None) for y in x if y in d1] for x in df1['keywords']]
print (df1)
                                keywords  \
0         [loans, mercedez, bugatti, a4]   
1      [trump, usa, election, president]   
2           [galaxy, 7s, canon, macbook]   
3  [beiber, spiderman, marvels, ironmen]   

                                     new  
0     [finance, mercedez, bugatti, audi]  
1                   [politics, politics]  
2                          [apple, dslr]  
3  [music, entertainment, entertainment]
jezrael
  • 822,522
  • 95
  • 1,334
  • 1,252