반응형
R로 워드 클라우드 만들기 (tidytext)- 도널드 트럼프 취임사
2018년 11월 21일
예전에 빅데이터라는 말이 이제 막 유행을 했을때 본인을 빅데이터 전문가라고 스스로 팔고다니는 사람들이 많았는데, 그 사람들이 방송에서 제일 많이 보여줬던 것이 이 워드클라우드라는 것이다. 워드클라우드는 그냥 비정형 데이터 종류의 하나인 텍스트 데이터를 의미단위(tokenize)로 구분해서 빈도 별로 가중치를 줘서 Plot을 하는 것에 불과하다. 통계학, 머신러닝 지식도 없이 이런거 만들고 빅데이터 전문가 소리를 들었으니… 참 돈 벌기 쉬웠을때였다 싶다. 잡설이 길어지니 이에 대한 썰은 나중에 쓰는 것으로 하자.
필요한 패키지
library(readr)
library(wordcloud)## Loading required package: RColorBrewerlibrary(tidytext)
require(RColorBrewer)
library(ggplot2)
library(wordcloud)
library(dplyr)##
## Attaching package: 'dplyr'## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, unionSys.setlocale("LC_ALL", "English")## [1] "LC_COLLATE=English_United States.1252;LC_CTYPE=English_United States.1252;LC_MONETARY=English_United States.1252;LC_NUMERIC=C;LC_TIME=English_United States.1252"텍스트 파일 로드
살펴볼 예제는 미국 도널드 트럼프 대통령의 취임사이다. 예제 파일 다운 tidytext 패키지는 영어밖에 지원을 안하고 한글로 텍스트 마이닝을 하려면 전희원님의 패키지를 써야한다.
mystring<-read_file("C:/Users/Sooyeong/Documents/text_mining/trump.txt")
print(mystring)## [1] "Chief Justice Roberts, President Carter, President Clinton, President Bush, President Obama, fellow Americans, and people of the world: thank you.\r\n\r\nWe, the citizens of America, are now joined in a great national effort to rebuild our country and to restore its promise for all of our people.\r\n\r\nTogether, we will determine the course of America and the world for years to come.\r\n\r\nWe will face challenges. We will confront hardships. But we will get the job done.\r\n\r\nEvery four years, we gather on these steps to carry out the orderly and peaceful transfer of power, and we are grateful to President Obama and First Lady Michelle Obama for their gracious aid throughout this transition. They have been magnificent.\r\n\r\nToday<U+0092>s ceremony, however, has very special meaning. Because today we are not merely transferring power from one Administration to another, or from one party to another <U+0096> but we are transferring power from Washington, D.C. and giving it back to you, the American People.\r\n\r\nFor too long, a small group in our nation<U+0092>s Capital has reaped the rewards of government while the people have borne the cost.\r\n\r\nWashington flourished <U+0096> but the people did not share in its wealth.\r\n\r\nPoliticians prospered <U+0096> but the jobs left, and the factories closed.\r\n\r\nThe establishment protected itself, but not the citizens of our country.\r\n\r\nTheir victories have not been your victories; their triumphs have not been your triumphs; and while they celebrated in our nation<U+0092>s Capital, there was little to celebrate for struggling families all across our land.\r\n\r\nThat all changes <U+0096> starting right here, and right now, because this moment is your moment: it belongs to you.\r\n\r\nIt belongs to everyone gathered here today and everyone watching all across America.\r\n\r\nThis is your day. This is your celebration.\r\n\r\nAnd this, the United States of America, is your country.\r\n\r\nWhat truly matters is not which party controls our government, but whether our government is controlled by the people.\r\n\r\nJanuary 20th 2017, will be remembered as the day the people became the rulers of this nation again.\r\n\r\nThe forgotten men and women of our country will be forgotten no longer.\r\n\r\nEveryone is listening to you now.\r\n\r\nYou came by the tens of millions to become part of a historic movement the likes of which the world has never seen before.\r\n\r\nAt the center of this movement is a crucial conviction: that a nation exists to serve its citizens.\r\n\r\nAmericans want great schools for their children, safe neighborhoods for their families, and good jobs for themselves.\r\n\r\nThese are the just and reasonable demands of a righteous public.\r\n\r\nBut for too many of our citizens, a different reality exists: Mothers and children trapped in poverty in our inner cities; rusted-out factories scattered like tombstones across the landscape of our nation; an education system, flush with cash, but which leaves our young and beautiful students deprived of knowledge; and the crime and gangs and drugs that have stolen too many lives and robbed our country of so much unrealized potential.\r\n\r\nThis American carnage stops right here and stops right now.\r\n\r\nWe are one nation <U+0096> and their pain is our pain. Their dreams are our dreams; and their success will be our success. We share one heart, one home, and one glorious destiny.\r\n\r\nThe oath of office I take today is an oath of allegiance to all Americans.\r\n\r\nFor many decades, we<U+0092>ve enriched foreign industry at the expense of American industry;\r\n\r\nSubsidized the armies of other countries while allowing for the very sad depletion of our military;\r\n\r\nWe<U+0092>ve defended other nation<U+0092>s borders while refusing to defend our own;\r\n\r\nAnd spent trillions of dollars overseas while America<U+0092>s infrastructure has fallen into disrepair and decay.\r\n\r\nWe<U+0092>ve made other countries rich while the wealth, strength, and confidence of our country has disappeared over the horizon.\r\n\r\nOne by one, the factories shuttered and left our shores, with not even a thought about the millions upon millions of American workers left behind.\r\n\r\nThe wealth of our middle class has been ripped from their homes and then redistributed across the entire world.\r\n\r\nBut that is the past. And now we are looking only to the future.\r\n\r\nWe assembled here today are issuing a new decree to be heard in every city, in every foreign capital, and in every hall of power.\r\n\r\nFrom this day forward, a new vision will govern our land.\r\n\r\nFrom this moment on, it<U+0092>s going to be America First.\r\n\r\nEvery decision on trade, on taxes, on immigration, on foreign affairs, will be made to benefit American workers and American families.\r\n\r\nWe must protect our borders from the ravages of other countries making our products, stealing our companies, and destroying our jobs. Protection will lead to great prosperity and strength.\r\n\r\nI will fight for you with every breath in my body <U+0096> and I will never, ever let you down.\r\n\r\nAmerica will start winning again, winning like never before.\r\n\r\nWe will bring back our jobs. We will bring back our borders. We will bring back our wealth. And we will bring back our dreams.\r\n\r\nWe will build new roads, and highways, and bridges, and airports, and tunnels, and railways all across our wonderful nation.\r\n\r\nWe will get our people off of welfare and back to work <U+0096> rebuilding our country with American hands and American labor.\r\n\r\nWe will follow two simple rules: Buy American and Hire American.\r\n\r\nWe will seek friendship and goodwill with the nations of the world <U+0096> but we do so with the understanding that it is the right of all nations to put their own interests first.\r\n\r\nWe do not seek to impose our way of life on anyone, but rather to let it shine as an example for everyone to follow.\r\n\r\nWe will reinforce old alliances and form new ones <U+0096> and unite the civilized world against Radical Islamic Terrorism, which we will eradicate completely from the face of the Earth.\r\n\r\nAt the bedrock of our politics will be a total allegiance to the United States of America, and through our loyalty to our country, we will rediscover our loyalty to each other.\r\n\r\nWhen you open your heart to patriotism, there is no room for prejudice.\r\n\r\nThe Bible tells us, <U+0093>how good and pleasant it is when God<U+0092>s people live together in unity.<U+0094>\r\n\r\nWe must speak our minds openly, debate our disagreements honestly, but always pursue solidarity.\r\n\r\nWhen America is united, America is totally unstoppable.\r\n\r\nThere should be no fear <U+0096> we are protected, and we will always be protected.\r\n\r\nWe will be protected by the great men and women of our military and law enforcement and, most importantly, we are protected by God.\r\n\r\nFinally, we must think big and dream even bigger.\r\n\r\nIn America, we understand that a nation is only living as long as it is striving.\r\n\r\nWe will no longer accept politicians who are all talk and no action <U+0096> constantly complaining but never doing anything about it.\r\n\r\nThe time for empty talk is over.\r\n\r\nNow arrives the hour of action.\r\n\r\nDo not let anyone tell you it cannot be done. No challenge can match the heart and fight and spirit of America.\r\n\r\nWe will not fail. Our country will thrive and prosper again.\r\n\r\nWe stand at the birth of a new millennium, ready to unlock the mysteries of space, to free the Earth from the miseries of disease, and to harness the energies, industries and technologies of tomorrow.\r\n\r\nA new national pride will stir our souls, lift our sights, and heal our divisions.\r\n\r\nIt is time to remember that old wisdom our soldiers will never forget: that whether we are black or brown or white, we all bleed the same red blood of patriots, we all enjoy the same glorious freedoms, and we all salute the same great American Flag.\r\n\r\nAnd whether a child is born in the urban sprawl of Detroit or the windswept plains of Nebraska, they look up at the same night sky, they fill their heart with the same dreams, and they are infused with the breath of life by the same almighty Creator.\r\n\r\nSo to all Americans, in every city near and far, small and large, from mountain to mountain, and from ocean to ocean, hear these words:\r\n\r\nYou will never be ignored again.\r\n\r\nYour voice, your hopes, and your dreams, will define our American destiny. And your courage and goodness and love will forever guide us along the way.\r\n\r\nTogether, We Will Make America Strong Again.\r\n\r\nWe Will Make America Wealthy Again.\r\n\r\nWe Will Make America Proud Again.\r\n\r\nWe Will Make America Safe Again.\r\n\r\nAnd, Yes, Together, We Will Make America Great Again. Thank you, God Bless You, And God Bless America."mystring<-data_frame(mystring)예제파일을 불러오고 프린트를 해보니 이렇게 지저분한 원 텍스트 데이터가 로드되었다.
의미단위 분절(Tokenize)
tidytext의 unnest_tokens 함수는 원래의 데이터를 받아들여 문장을 tokenize를 해 주는 기능을 한다. 이 함수를 이용하여 의미 분절을 하고 난 뒤에 의미없는 단어들(stop_words) 안티조인으로 없애준다. stop_words에 포함되는 단어는 The, I, we, a 등등을 포함한다. 이제 wordcloud를 그리기 위한 전처리 과정이 끝났다.
아래 그래프는 트럼프의 연설에서 3번이상 언급된 단어들이다. America가 관련 단어가 압도적으로 많고 Power, Wealth 처럼 경제력 국방력을 직접 언급한 것이 인상적이다.
#함수 첫 번째요소 - 아웃풋, 두 번째- 인풋
tidy_trump<-mystring %>%
unnest_tokens(word,mystring)
tidy_trump_nostop<-tidy_trump %>%
anti_join(stop_words)## Joining, by = "word"tidy_trump_nostop %>%
count(word, sort=TRUE) %>%
filter(n>3) %>%
mutate(word=reorder(word,n)) %>%
ggplot(aes(word,n))+
geom_col()+
xlab(NULL)+
coord_flip()
워드클라우드 (Word Cloud) 그리기
#칼라 설정
colr <- brewer.pal(8,"Dark2")
tidy_trump_nostop %>%
count(word,sort=TRUE) %>%
with(wordcloud(word, n, max.words=30, colors=colr))
이상으로 트럼프 취임사를 예제를 가지고 워드클라우드를 만들어 봤다. 이는 정말 간단한 예제이며 더 관심이 있는 분은 Text Mining with R을 참고하시길 바란다.
반응형
댓글