Data Analysis

Set the system path for Python Jupyter notebooks

Posted on

In Jupyter, when I was a newbie, I often needed to reference some Python library code located in some weird place on my PC, so I did this at the top of each Jupyter notebook I created:

import sys
sys.path.append(‘C:\users\name\code\my-Python-object-location’)

Doing so made the path (temporarily) part of sys.path for as long as that session was active. But when I started a new notebook, I always had to include sys.path.append() again at the top of each new notebook. Drove me nuts.

Here’s the fix:

Add your Python object path(s) to “PYTHONPATH” or an exiting “path”  entry in your system environment variables (via the Windows Control Panel).

How to do it:

On your system (for Windows 10, enter the following in the “Type here to search” box, screen bottom left), search for “control panel” then in the upper right of the panel, search for “environment” and click on “Set your environment variables”

Next, in the Environment Variables section (see image below), check if you already have PYTHONPATH. If yes, select it and click “Edit” and add additional paths as needed. If it’s not there, click “New” and add PYTHONPATH (if you have an existing ‘path’ variable, simply edit it. But I like to add PYTHONPATH to keep it logically separate from the generalized Windows system ‘path’ variable).

Paths in environment variables such as PYTHONPATH need to be separated with a semicolon, “;” … like this: ‘C:\users\name\code\my-library111′;’C:\users\name\code\my-library222′;’C:\users\name\code\my-library333’

So, click ‘Save’ farther down at the bottom of the Environment Variables box and you are done.

Remove the sys.path.append() code from your notebooks and restart them and you should be good to go. (Just to be safe, adjust one notebook first and check it out to make sure this system path fix is working for you!)

Good luck. The game is afoot!

 

add python path to system Peter Bakke

Astronomy

Mars Spacecraft was Lost in Translation

Posted on

NASA lost its $125-million Mars Climate Orbiter because spacecraft engineers failed to convert from English to metric measurements when exchanging vital data before the craft was launched, space agency officials said.

A navigation team at the Jet Propulsion Laboratory used the metric system of millimeters and meters in its calculations, while Lockheed Martin Astronautics in Denver, which designed and built the spacecraft, provided crucial acceleration data in the English system of inches, feet, and pounds.

As a result, JPL engineers mistook acceleration readings measured in English units of pound-seconds for a metric measure of force called newton-seconds.

In a sense, the spacecraft was lost in translation.

Mars Climate Orbiter

From 1998:

Source:  http://articles.latimes.com/1999/oct/01/news/mn-17288

Data Analysis

Do I Love or Hate R’s knitr?

Posted on

Depends. I’m sharing my love/hate experience with knitr. When it works, it’s divine. When it doesn’t, it’s diabolical. knitr is finally working well in my local PC’s RStudio, but I find that I’m running knitr every hour or so to make sure I have not introduced something into my .rmd file that breaks it. I had wrongly assumed that knitr basically takes a “screen capture” or the like of my .rmd file and outputs the page as html, PDF, etc. In fact, knitr executes EVERYTHING in my ever-growing project .rmd file and if, say, a variable is undefined or I have an R code chunk that has the same name, knitr will barf and halt execution resulting in nada, nil, zip, nothing, nichevo. That’s why I run knitr several times a day, and good thing that I do because it catches stuff. I don’t want to wait for my project to be complete only to find that I have to spend hours fixing syntax errors (whatever) that knitr so gleefully finds. Onward.

knitr-peter-bakke

 

Uncategorized

Setting an (almost) unique seed for a random generator

Posted on

In Data Analysis and other programming endeavors, we frequently have to set a random() seed to, say, select a sample of observations from a very large dataset.

You might want to select a 10,000 sample from a million observation dataset. Good idea to avoid crashing your PC. You should use a random seed to do so.

If you truly want an (almost) unique random seed, try my tried and true method (from my IBM software engineering days) of using date/time … preferably utilizing milliseconds.

Example:  a seed of 1471300214792 milliseconds converted to date/time is 16 August 2016 01:30:14:792. Set your seed to today’s date/time in milliseconds.  You’ll never see that particular programmatical random seed in this lifetime (perhaps only if 1 million monkeys tapped on a calculator for 100+ million years… or the time it takes for Elon Musk’s Falcon Heavy-launched Tesla to fall into the sun ). Think about it.

Have fun, brainiacs.

 

                                         Wiki
Journalism

Robert Frost found poetry in New Hampshire

Posted on

Hat’s off to Mike Pride for writing the following article from the Concord Monitor “NH 100” biography series published at the turn of the millennium. It deserves to see the light of day. Enjoy.

During his Derry, NH years, Robert Frost found his voice

Sunday, March 28, 1999

By MIKE PRIDE
Monitor editor


Farming and teaching shaped the man into the nation’s most beloved poet.

While living on a farm in Derry during the first decade of the 20th century, Robert Frost discovered his poetic voice. He was an outsider – a Californian by birth – and in time fame, fortune and a restless spirit lured him elsewhere. But no matter where he wandered, his voice, like his name, fixed the maddening character of New Hampshire in American lore.

His years in Derry tested Frost’s own character. Because he suffered chronic fevers, chills, chest pains and breathing difficulties, a physician had warned him against the sedentary life of writing and teaching for which he had prepared. In 1900, the year he moved to Derry, his first-born son died of cholera and his mother died of cancer at a sanatorium in Penacook. Frost was just 26 years old, but life seemed to mock his desire to be a poet.

To make a living in Derry, Frost first farmed, then taught. He was a moderate success as a farmer, yet he bore the whispered stigma of “failed henman” into the classroom at Pinkerton Academy. As a teacher he excelled, urging on his students an approach to writing that had already become central to his poetry: Write about common experience in an uncommon way. But a career in education was not for him. “Why have only your labor for your pains?” he wrote, lamenting that teaching stole from his poetry.

What Frost discovered in Derry was genuine speech – American speech – and a method of posing that speech against the formal meter of poetry.

“He wants two things at once: the sound of speech and absolute metrical regularity,” said Donald Hall, the Wilmot poet who himself moved to a New Hampshire farm 75 years after Frost. Another rural New England poet, Wesley McNair, said that in rebelling against the sing-song of 19th-century verse, Frost “turned to the life and energy of spoken English to renew poetry.”

To renew poetry: Frost’s ambition was no less than that. Five years after selling the farm, he summed up his time in Derry when he told a Boston reporter: “I kept farm, so to speak, for nearly 10 years, but less as a farmer than as a fugitive from the world that seemed to disallow me. . . . I can see now that I went away to save myself and fix myself before I measured my strength against all creation.”

In Derry, Frost wrote or drafted many of his finest poems. After Derry, the years of his long life only deepened the ironic Yankee voice that he borrowed from the neighbors. It is a voice that, to this day, calls its listeners to think at least twice about what is being said.


A new beginning

In October 1900, the weekly Derry News told its readers: “R. Frost has moved upon the Magoon place which he recently bought. He has a flock of nearly 300 Wyandotte fowls.”

In fact, it was Frost’s grandfather who had bought the farm, for $1,700. The Frost family – Robert, his wife Elinor and their 1-year-old daughter Lesley – had been evicted from a farm they rented in Methuen, Mass. The poultry farming had begun well enough in Methuen, but the death of 3-year-old Elliott Frost brought despair upon the young couple, and Frost fell behind in the rent.

The new farm, a white clapboard house, a barn and an apple orchard on 30 acres, promised a new beginning. If Frost managed to hang on to the farm for 10 years, his grandfather said, it would be his. The grandfather also gave Frost an annuity of $500, which would grow to $800 after 10 years, but he attached a condition to his generosity. He did not trust Frost to go it alone as a farmer and insisted that Carl Burell move in and help.

Burell, who had been a high school friend of the Frosts in Lawrence, Mass., worked in a box factory in Pembroke, but he was a literate man with a passion for flowers. Four years earlier, he had found the cottage in Allenstown where Robert and Elinor took their belated honeymoon. They stayed the summer, and Robert saw Burell often. In garden walks and through his discourses on the cycles of nature, Burell nurtured the interest in botany that would soon blossom in Frost’s poems.

On the Derry farm, Burell’s contribution was more practical. For a year and a half, he and his 84-year-old grandfather, Jonathan Eastman, lived on the second floor and took meals with the Frosts. Burell tended the cow, the horse, the apple and pear trees and the vegetable garden. He also helped Frost make hen coops.

The poultry farming was Frost’s work. He was a marginal success as a farmer, although he would later discount both his efforts and the results. Eggs and apples were his cash crops, and he managed to keep his growing family fed.

Elinor gave birth to a son and two daughters during the first five years at Derry. Both parents were teachers, and they taught their children at home. Most of this duty fell on Elinor, but Robert was an attentive father who read to his children, taught them botany and astronomy and went over their writing with them.


A fast friend

Frost’s life in Derry was full of contradictions. He rose late by a farmer’s standards, a habit that caused friction between him and Burell, but he also stayed up into the night writing poetry at the kitchen table. He felt alienated from the literary world of Boston and New York, yet he developed fast friendships with his fellow poultrymen.

“I was never really out of the world for good and all,” he later wrote. “I liked people even when I believed I detested them.”

As a young girl, Frost’s daughter Leslie went on long walks and errands with her father. She quickly learned that his love of conversation meant they never got home on time. An acquaintance once said of Frost: “He talks all day and every day.” But he was a listener, too, and his keen ear gleaned both the manner of speech and the mastery of subject that transformed his poetry.

Frost’s closest friend was a poultryman named John Hall, 30 years his elder, from neighboring Atkinson. Hall was a salty, down-to-earth man who raised prize fowl. Frost admired and learned from Hall’s farming expertise, but Hall had a more vital lesson to teach. Frost recognized Hall’s speech as “homely, shrewd and living,” and he paid close attention to it. Before long, he had concluded that “real artistic speech was only to be copied from real life.”

In 1903, to supplement his income, Frost began writing fictional sketches for $10 apiece for two poultry trade magazines. A recurring character was modeled on Hall, and the speech patterns in the dialogue closely resembled the ones that Frost had begun to shape into poems. Sometimes in the sketches, the characters spoke directly to one another with little commentary or context from the narrator – the same technique Frost used in such talking poems as “The Death of the Hired Man,” written a few years later.

In that poem, a farmer’s wife tells him that a poor old bum has just returned in search of work, though it is wintertime and there is little extra to do. She beseeches her husband to be kind to the man. The husband replies:

“When was I ever anything but kind to him? But I’ll not have the fellow back,” he said.”I told him so last haying, didn’t I? If he left then, I said, that ended it. What good is he? Who else will harbor him? At his age for the little he can do? What help he is there’s no depending on. Off he goes always when I need him most. He thinks he ought to earn a little pay, enough at least to buy tobacco with, So he won’t have to beg and be beholden. ‘All right,’ I say, ‘I can’t afford to pay Any fixed wages, though I wish I could.” Someone else can.’ ‘Then someone else will have to.’I shouldn’t mind his bettering himself If that was what it was. You can be certain, When he begins like that, there’s someone at him Trying to coax him off with pocket money, -In haying time, when any help is scarce.In winter he comes back to us. I’m done.”

Frost absorbed more than the language in and around Derry; rural New Hampshire gave him settings and subjects that lasted him a lifetime. “I was interested in neighbors for more than merely their tones of speech,” he wrote in retrospect. He also found himself interested in “gossip for its own sake.”

Poetry was all around him – in conversation, in the woods, in the everyday lives of Yankee farmers.


Teacher

To feed his large family, Robert Frost took a job teaching English at Pinkerton Academy in 1906. He was an unconventional teacher but a superb one. Although the job took time and creative energy away from his writing, it also helped him to think through his ideas about poetry. Teaching enforced a new regimen on Frost. No longer could he write into the night after the children were in bed, then sleep late. He had to grade papers and prepare for class, and he had to rise early and walk the two miles to school each day.

In an era in which Victorian primness governed the dress and demeanor of teachers, Frost behaved otherwise. He always entered the classroom “at a gallop,” a student recalled. “His hair, cut at home, was blown in all directions by the wind. . . . His clothes were rumpled and ill-fitting.” Before the class, he “would slump down in his chair behind his desk, almost disappearing from sight except for his heavy-lidded eyes and bushy brows.”

Whatever his posture, Frost was an engaging, vigorous and demanding teacher. In addition to a heavy reading load, his freshmen and sophomores had to write or present 50 themes per academic year. He expected correct grammar and spelling. He taught the usual books, but his purpose and his method were singular. Above all, he pushed his students to have ideas of their own. In the classroom he “was seeking kindred spirits – to comfort them and to comfort me,” he said in retrospect. For him, the joy in literature occurred between the reader and the book – and the joy was the point. The pleasure of words, read aloud or recited, had a value all its own.

Frost’s superiors noticed his talents. He was soon writing Pinkerton’s English curriculum, stating its twofold purpose in the very first sentence: “to bring our students under the influence of the great books, and to teach them the satisfaction of superior speech.” New Hampshire’s superintendent of public instruction observed Frost’s classes and admired his innovation so much that he invited him to speak on his methods at a state teachers’ convention.

Over time, teaching displaced farming as Frost’s livelihood. His annual salary climbed to $1,000, supplemented by the annuity from his grandfather. He and his family left the farm and moved to Derry Village, and it wasn’t long before his success at Pinkerton led to a new opportunity. The Frosts sold the Derry farm in 1911. The next year, they headed north so that Robert could take a job at Plymouth Normal School.

Yet what was true of his farming was true of his days in the classroom: Frost was a teacher out of necessity. He was fond of telling his students they should not write unless they had something to say, and if they didn’t have it, they should go get it. Farming and teaching were the means through which Frost had gone and gotten what he needed. Now he was ready for his calling: poetry.


Poet

In January 1901, three months after moving to Derry, Frost sent a few poems to Susan Hayes Ward, a magazine editor in New York. They were, he wrote to her, a “selection from the poems I have been writing with a view to a volume someday.” During the next five years, from the ages of 27 to 32, he sent no poems to any editor.

Though flashes of the future Frost occur in his early poems, many of them come directly out of the 19th-century English tradition. Here is the first stanza of “Stars”:

How countlessly they congregate

O’er our tumultuous snow,Which flows in shapes as tall as trees

When wintry winds do blow!-

The Derry years were a period of incubation for Frost. He wrote poems – and good ones – during these years, but his real achievement was to make his way as a poet. He found plain speech and real life to put in his poems, and he developed a theory about how to do it.

This breakthrough he himself described a few years after leaving Derry in letters to John Bartlett, the valedictorian of the class of 1910 at Pinkerton Academy and a favorite student of Frost. The idea, Frost wrote, was that the sounds – not just the words – of spoken sentences conveyed meaning to the ear. If a person began a sentence, “My father used to say -” or said, “Put it there, old man,” the ear was practiced in receiving the natural rhythms of the sentence. It could take the meaning without hearing the words.

“A sentence is a sound in itself on which other sounds called words may be strung,” Frost wrote Bartlett. “A man is all a writer if all his words are strung on definite recognizable sentence-sounds. The voice of the imagination, the speaking voice must know certainly how to behave . . . in every sentence he offers.”

Literature, to Frost, was not a matter of presenting readers with some new truth or discovery. “It is never to tell them something they don’t know, but something they know and haven’t thought of saying,” he wrote. “It must be something they recognize.” In a journal he kept a few years after the Derry period, he wrote: “A poem would be no good that hadn’t doors. I wouldn’t leave them open though.”

Placing himself as a poet gave Frost a direction for his work. Like his traveler who comes to two roads in a yellow wood, he embraced this direction as his fate. Nearly all of Frost’s best poems – even many written years later – flow from the Derry period.

In “Mending Wall,” written shortly after he left New Hampshire, almost any short passage serves to illustrate how he set down in iambic pentameter the sentence sounds of the speech he had so carefully absorbed as a chicken farmer in Derry. Or, as Donald Hall put it, how Frost went about “breaking the American sentence against the English line.” Here the farmer narrating the poem is talking about setting stray boulders back in place in a stone wall:

We have to use a spell to make them balance:

“Stay where you are until our backs are turned.”

We wear our fingers rough with handling them.

Oh, just another kind of outdoor game,

One on a side. It comes to little more:

There where it is we do not need the wall:

He is all pine and I am apple orchard.

My apples will never get across

And eat the cones under his pines, I tell him.

He only says, “Good fences make good neighbors.”

Although the subject matter is rustic, the sentences in this poem sound as fresh as they did nearly nine decades ago, when Frost wrote them. The reader knows from the form and cadences that this is poetry, but it is also plain speech.

Through context and through writing for the American ear, Frost made the last line quoted mean something different from what the speaker intended to say. This is Yankee irony boiled down to the bone: The stone wall has outlived its use, yet the narrator’s stubborn neighbor clings to the old saw about good fences making good neighbors.


After Derry

Robert Frost understood stubbornness. Despite the ailments that led to his move to Derry, he lived to be 88 years old. Once he had achieved his ambition and become the great American poet of the 20th century, he jealously guarded his place at the top of the heap. He made light of his farming years in Derry, but he cultivated his image as a flinty but lovable old Yankee.

After a short time teaching at Plymouth, Frost left New Hampshire to take his family to England. A Boy’s Will, his first book of poems, was published there in April 1913. He had just turned 39 years old. His New Hampshire years had been so fruitful that he had many more poems ready for publication, and his second book followed soon after the first.

When he returned to the United States in 1915, the literary crowd in Boston and New York already knew of his successes in England. He was soon able to land university jobs with few teaching responsibilities, giving him the time he needed to write poetry.

Frost owned a farm in Franconia in the late teens but for the sake of his livelihood spent most of his life near universities. By the 1920s, he had chosen Vermont, not New Hampshire, for his rural home.

His poetry has universal appeal, but all New Hampshirites who read Frost closely recognize in his poems people they know, if not some part of themselves. Frost’s imagery – the birches and brooks, the stone walls, leaf-covered trails and snowy woods – are real and familiar. And so is the voice in which Robert Frost wrote, the voice he found as a poultryman in Derry.

Sunday, March 28, 1999

 

Data Analysis

How to plot a variable in R that has spaces it it?

Posted on

There are thousands of datasets on the web available for analysis using R. Many of them are listed by the plus or minus 175 countries, like “United States” or “Cote d’Ivoire”

So, ignoring for a moment that the experts say never name a variable with spaces, in the real world how do you plot a variable with spaces in its name?

Simple. When programming, encase the variable name with backticks. Like so: `United States`

Example: see below the line with, y=`Costa Rica`

ggplot(df, aes(Year, group=1)) +
ylab(“Country”) +
geom_line(aes(y = `Costa Rica`, color=”Costa Rica”)) +
geom_point(aes(y = Belgium, color=”Belgium”))

r-Peter_Bakke

 

Journalism

Archive of Concord Monitor Online. Party like it’s 1999.

Posted on

Check out this archive of Concord Monitor Online (Concord, NH) when I ran the show there 1997-2005 as online manager. Don’t laugh too much at the retro-1999 layout. What I’d like you to see is that the Monitor’s content at the turn of the century (like the ‘NH Top 100’) was really remarkable. Many thanks to the likes of editors Mike Pride and Mark Travis. Let’s not forget all the staff writers who wrote the biographical articles on some amazing New Hampshirites.
https://web.archive.org/web/19990429230127/http://www.concordmonitor.com:80/index/top100.shtml

 

Peter Bakke Online manager Concord Monitor, Concord, NH 1999

Data Analysis

Unable to install packages in R

Posted on

Running Rstudio on Windows for more than a day or so without shutting down? Are you now getting errors installing packages? I suggest Windows has “lost” its mind. Save and Restart #rstudio & #windows. Likely your problems will go away.  #DataAnalytics #DataAnalyst #Statistics

Searches related to this R (RStudio) package installation problem that might provide solutions as well:

Moi

Once upon a time, I may have been a pig farmer. And there is nothing wrong with that.

Posted on

You know, whenever you hear about someone claiming that they are reincarnated from a previous life, the following seems to be true: they once were a prince, princess, queen or king. Or maybe a duke, duchess, earl, shaman. Whatever.

I am proud to announce right here and now that I, if I am indeed reincarnated, was probably a pig farmer in a previous life somewhere in medieval Germany.

Hey, bacon had to come from somewhere.

Now top that, you grandees.

Data Analysis

Are You Having Trouble Plotting a Time Series in R? Here’s one solution.

Posted on

Context: I was using Gapminder.com data, so the data is presented in csv format.
The data consists of various country-related data with rows as country names and columns as years.

Step 1. Read into a data.frame the local .csv file that you downloaded from the Gapminder website.

Note that “row.names = 1” saves each of the 175 or so country names in the rows. Columns will be years.

df <- read.csv(‘path.to.your.local.file.here’, header = T, row.names = 1, check.names = F)

Step 2. Transform the data.frame. 

Years are now rows instead of columns. Country names are now cols.

df_t <- t(df)

Step 3. Add a “Year” column created from the row.names. For me, this was the missing link for
successfully plotting the data, else I was floundering.

Reason for this: We need this col to be numeric variables in order to plot
…which are the years in the time series

df_t$Year <- row.names(df_t)

Step 4. That’s it! Now you can plot. It’s that simple.

For example:

ggplot(df_t, aes(Year, group = 1)) +
geom_line(aes(y = Albania, color=“Albania”)) +
geom_point(aes(y = India, color=“India”))

Etc…

Peter Bakke - Data Analyst Gapminder.com plotting time series