To the left, looking north on Main Street from the corner of Curtis, was a long double row of American Elms linking Main and Abel Streets. These were planted in the 1870s by John O'Toole along the driveway to his 20 room Victorian mansion. O'Toole grew produce, hay, and raised thoroughbred race horses here and on acreage near Coyote Creek.

The mansion and the 100+ acres of farmland surrounding it were bought by James Boyd in 1883. He did not keep it for long. In 1884, he sold the property for the then grand sum of $24,000 to Santa Clara County for use as an almshouse. Newspapers of the day called it a "palace for the poor," and other such terms to voice their indignation over what they saw as a waste of public money.

In spite of initial negative press reaction, it was here the county's poor and homeless could find shelter and work on the farm. A portion of what was grown was given to the poor to sell. Many residents of the almshouse set up stands along Oakland-San Jose Highway (now Main Street) to sell produce and wild mushrooms to travelers. In this way those temporarily down and out could earn their keep and put aside a little to get them back on their feet. Poverty was not viewed as a permanent condition in those days. It was believed and practiced that anyone who was willing to work hard could get ahead.

In the 1940s the county began to house low risk prisoners at the site. Over time, more and more of the ranch buildings were torn down to make way for cell block construction. In 1962, eight elderly men, the last Almshouse residents, were evicted and the historic Victorian mansion was demolished to make room for increasing numbers of prisoners to be housed at the County Jail. The name of the jail became Elmwood because of these stately trees. Today, there remains only one of the buildings from the agricultural days of Elmwood's history left standing. It is called the "library" and public access is prohibited.

In the late 1970s the Fire Chief, alarmed over a large branch breaking off in a wind storm, ordered all of the trees topped and stripped of their leaf bearing branches. The upper third of each tree was removed. Although nearly all of the ancient arbor survived the assault, the cuts were left parallel to the ground. Rainwater puddling on this surface then caused rot and allowed insects to invade the trees.

With new housing developments being constructed to the north and south of the O'Toole Elms, The city council decided that the trees would have to be destroyed.  Although citizens wrote letters and lobbied to try to preserve at least one of the historic trees, none were spared.

The last bridge over the old Penitencia Creek bed, a 15 foot wide one lane culvert, was located at the east end of the trees just a few feet from Main Street. It had been the entrance to the driveway of the Alms House and later to the jail.  The date "1910" was embossed on the concrete abutment of the bridge. The Milpitas fire department installed training facilities around and atop the bridge with high cyclone fencing keeping out the public. Later, the bridge was quietly destroyed.  No one can now experience sitting on the concrete abutments like the poor of a past century did while selling their vegetables.