Quality Data, Quality Decisions: Why Web Scraping is Essential for Advanced Analytics
Gediminas Rickevičius·9 min
Capitalism processes data by directly connecting all producers and consumers to one another, and allowing them to exchange information freely and make decisions independently.
Thus, laissez-faire capitalism distributed the work of analyzing data, teasing out patterns, and making recommendations and decisions based on the obtained information between a plethora of independent but very closely interconnected and actively interacting processors.
[caption id="attachment_15447" align="aligncenter" width="2500"]
Photo by Aditya Vyas on Unsplash. New York stock exchange — personification of distributed data processing.[/caption]
A myriad actors interact with each other and determine the course of events. As an Austrian economist, F. Hayek put it,
"… in a system in which the knowledge of the relevant facts is dispersed among many people, prices can act to coordinate the separate actions of different people in the same way as subjective values help the individual to coordinate the parts of his plan."
Exactly due to data-processing reasons advocates of a true capitalism favor lower taxes.
When taxes are low, individuals retain most of the money and therefore have greater opportunities to effectively invest these resources. When taxes are increased, this means that more capital accumulates in the state’s coffers’ and therefore the government makes most of the decisions. But it is known that the government is not as efficient as private investors; thus, in a distributed process of decision-making, the allocation of resources is performed much better than in centralized.
The Soviet science ministry forced all Soviet biotech laboratories to adopt the theories of Trofim Lysenko — the infamous head of the Lenin Academy for Agricultural Sciences. Lysenko rejected the dominant genetic theories of his day. He insisted that if an organism acquired some new trait during its lifetime, this quality could pass directly to its descendants. This idea flew in the face of Darwinian orthodoxy, but it dovetailed nicely with communist educational principles. It implied that if you could train wheat plants to withstand cold weather, their progenies will also be cold-resistant. Lysenko accordingly sent billions of counter-revolutionary wheat plants to be re-educated in Siberia — and the Soviet Union was soon forced to import more and more flour from the United States.If a company in the capitalist country decided to adopt Lysenko’s pseudoscience approach to agriculture, it would inevitably fail, but, unlike in the Soviet Union, other data processors would fill the gap, thereby mitigating the devastation inflicted by the failure of one of the processors. In the 20th century, distributed data-processing worked better than centralized because it was quick to adapt to changing conditions. When all available information is analyzed and decisions are made by a single decision-maker it is very hard to undertake the right measures and live up to the standards of the new technological era. Hence, the secret of capitalism’s success is the inherent efficiency of the distributed process of decision-making and the ability to adapt to changing conditions and adopt innovation. This is why centralized systems fail, while open and flexible and decentralized ones succeed.
Real-time institutional flow data and trading signals for serious investors.
Explore DataDrivenAlpha →Instantly repurpose any DDI article into a professionally produced short-form video.
Try DDI Media →