gameoversite.com

Wednesday, February 23, 2005

On Monday, at 10.14 pm UTC, the Wikimedia server cluster experienced a total power failure, taking down Wikipedia, Wikinews, and all other Wikimedia websites.

The servers are housed in a colocation facility (colo) in Tampa, Florida, USA. They occupy two racks, with each rack receiving electricity from two independent supplies. However, both supplies have circuit-breakers in them, and both opened at the same time, leading to a total power failure. All computers immediately went down. It’s normal for fire safety regulations to prohibit uninterruptible power supplies in colos, with the colo providing its own UPS and generator instead. The circuit breakers were on the computer side of this emergency power system, so none of the computers continued to receive power to survive the breaker trip or shut down safely.

The actual reason why the circuit breakers tripped is currently unknown.

When power was restored, it was discovered that most of the MySQL databases that store the data which makes up Wikipedia et al had been corrupted. The main database and the four slaves had all damaged the data on their hard disk drives beyond the ability of the auto-correction to repair. Only one copy survived safely, on a machine that is used for report generation and maintenance tasks, which remained 31 hours backlogged while catching up after an unusually heavy update load during the previous week.

Volunteer Wikimedia engineers worked through the night rebuilding the databases from the sole good copy onto the other servers. The Wikipedia database is over 180Gb in size, making the copying process last 1.5 – 2 hours for every server it was performed upon.

Regular back-ups of the database of Wikipedia projects are maintained – the encyclopedia in its entirety was not at risk. The last database download was made on February 9; all edits since then could only have been laboriously rebuilt from logs and recovered from the damaged database requiring much more time and effort.

Limited read-only service was established late Tuesday afternoon, with editing becoming possible 24 hours after the power failure. Final repairs continue now, as well as upgrades to prevent similar issues in the future. Server-intensive features, such as categories and ‘watchlists’ that display recent changes to selected articles to registered users, remain disabled to ease the load on the recovering systems.

The process which led to the damage originated with the operating system, disk controllers, or hard drives failing to flush the data correctly.

If the power to a database server is cut mid-write, the database may be corrupted and unreadable, however the operating system, hardware, and software are designed to make this very unlikely. In a previous incident in 2004 power was also lost to a server but the database was undamaged.

To avoid such damage, each database server saves a copy of an edit to be applied to the database on a separate storage system before making the actual update to the database itself. This so-called ‘write-ahead logging’ should ensure that in the event of a system crash, the database can be rebuilt from a ‘last-good’ state by replaying the edits saved in the log.

Earlier this year popular blogging site LiveJournal suffered a similar power failure when another customer at their colocation facility pressed an Emergency Power Off button, intended for use only by firefighters. The company suffered database corruption similar to that seen at Wikimedia.

LiveJournal are now fitting UPS to their servers to ensure that they have time to shut down safely in the event of a power failure. Wikimedia was said to be investigating the possibility of fitting similar equipment at the time of this failure.

Several pundits have suggested that the use of another database, such as the proprietary database Oracle or the free PostgreSQL, would have avoided the database corruption seen at the server cluster. A post-mortem of the incident show the failure was in the operating system, or the hardware, or some combination of the two. LiveJournal, which also uses MySQL, reported similar database corruption after their power cut.

The Wikimedia foundation only allows the use of free software on its systems, and future versions of the Mediawiki software will support the PostgreSQL database.

Users are reminded that during times of system failure or excessive demand, they can still search Wikipedia using Google. The articles may be viewed using Google’s cache.

Monday, October 26, 2020

On Friday, code hosting and sharing website GitHub blocked the public access to youtube-dl, a software which can download videos from the internet via the command-line. The blockade came after GitHub received a Digital Millennium Copyright Act (DMCA) take-down notice from the Recording Industry Association of America (RIAA). After stripping the metadata from the notice, GitHub published the take-down notice on their site.

Initially started in July 2008 by Ricardo Garcia, youtube-dl is a script written in Python which can download videos from multiple websites including YouTube, LiveLeak and Vimeo. youtube-dl is a FLOSS software and is under public domain. Currently, the repository on GitHub is locked for viewers other than maintainers of the project.

RIAA’s DMCA notice alleged the script’s purpose of existence was to “circumvent the technological protection measures used by authorized streaming services such as YouTube” and “reproduce and distribute music videos and sound recordings owned by our member companies without authorization for such use”.

youtube-dl has multiple unit tests in its source code, which test whether the software works in different circumstances or not. Some of the tests include checking if the script can download Creative Commons licensed videos, videos which did not have square pixels, videos with no age restriction, “offensive to some audiences” per YouTube community and age-restricted videos. One of the tests included the URL of some copyrighted songs. Citing this test, RIAA’s take-down notice claimed “comments in the youtube-dl source code make clear that the source code was designed and is marketed for the purpose of circumventing YouTube’s technological measures”.

RIAA’s notice published by GitHub alleged violation of 17 U.S. Code § 1201 Circumvention of copyright protection systems which says “No person shall circumvent a technological measure that effectively controls access to a work protected under this title”. RIAA listed a number of forks of youtube-dl and requested GitHub via the notice they all be made inaccessible.

The notice did not list any incident of anyone using youtube-dl to download or share copyrighted material, nor mention any damages that actually occurred. Unremarked by the notice, YouTube allows videos to be licensed under a Creative Commons Attribution license. When a copyright holder chooses to release their work, be it a photograph, a video, or audio, under a Creative Commons Attribution (CC BY) license, they allow everyone to freely own, share or modify the work as long as the reusers properly attribute the author of the work. YouTube also hosts many audio and video recordings in the public domain which can be used for any purpose without any restrictions.

youtube-dl is used by thousands of people around the world. Multiple Creative Commons-licensed and public domain videos on Wikimedia Commons are uploaded via a tool called video2commons, which relies on youtube-dl to download media. youtube-dl also lets users download videos from LiveLeak — a video-sharing platform for citizen journalism. Videos downloaded using youtube-dl are also used for the purpose of fair use, or for evidence.

youtube-dl comes with a small JavaScript interpreter where it acts as a web-browser would behave while receiving video data from the server. The script has “extractors” for various websites to handle videos from different sources. Whenever something is displayed on the user’s screen, the device has a copy of that content. Web browsers “download” data while surfing the web, though most of it is not persistent on the device. It is possible to download copyrighted photos by using a web-browser. The way the world wide web works, there are no technological prevention measures to prevent recording and sharing of content such as RIAA talks about in the notice. Photos and texts can be downloaded by taking screenshots, videos by screen recording tools, and audio by recording on a tape if not an audio recording software.

Multiple users expressed their disappointment on Twitter and Internet Relay Chat. One of the users said “this is yet another example of why we should use git as it was intended, as a distributed network, rather than rely on one single proprietary server”. Git is decentralised version-tracking software which is used by a large number of software companies and projects. It is possible to host one’s own git server for software development. While Microsoft’s GitHub is a centralised git server, development of software using git does not require a GitHub account.

Soon after the public access to the repository was locked, multiple users started sharing the source code via self-hosted git servers, Tor sites and via the Torrent protocol leading to a Streisand effect. Streisand effect is when a measure to censor information causes further spread of that information. The binary files of the software are still available on its website for users to download. Some people came up with esoteric ways to share the source code, by converting the compressed code into photographs and providing shell commands to convert to the source code.

GitHub’s DMCA repository, where the takedown notice was published for public viewing, was subject to contant vandalism from multiple GitHub users. One user submitted a pull request, merging the source code of youtube-dl along with the DMCA repository. This enabled users to view youtube-dl’s source code from within the DMCA repository, provided they know the commit id.

The Electronic Frontier Foundation said on Twitter “Youtube-dl is a legitimate tool with a world of a lawful uses. Demanding its removal from Github is a disappointing and counterproductive move by the RIAA.” Richard Stallman, the founder of the Free Software Foundation, has been highly critical of DRM (digital rights management, the subject of the DMCA) for many years now.

Wikinews reached out to Sergey M?, one of the maintainers of youtube-dl script, however Sergey said he “won’t give any comments at this time”. Later, he shared an update on the IRC channel. Sergey said, “they require complete removal of so called YouTube’s rolling cipher implementation […] GitHub requires in order to reinstate the repo […] under this conditions I could reinstate it in Saturday/Sunday already but this is an unsatisfactory outcome”. He also said, “I can’t guarantee whether [or] not we will bend over them considering the situation with @phihag [Philipp Hagemeister] but we’ll see soon what we can do in order to keep the max we have and mitigate potential legal issues at the same time”.

EFF is yet to respond to Wikinews queries. Wikinews also reached out to Philipp Hagemeister, a former maintainer and contributor of the youtube-dl project to discuss this takedown.

When did you get to know about the takedown notice and what were your initial reactions?

((Philipp Hagemeister)) I saw the takedown notice along with anyone else, on reddit. Since I am no longer involved with the youtube-dl project (except for occasional contributions, my maintainership ended in 2016), I don’t know any details.

((RS)) Does YouTube implement DRM for videos not under Creative Commons license, and if so, how does youtube-dl bypass it? Could you please elaborate the procedure?

((Philipp Hagemeister)) YouTube implements DRM for YouTube Movies. youtube-dl does not support those.

YouTube has multiple non-DRM video delivery protocols. I’m not up to date about specifics; my last dabbling in this was in 2015.

One of these protocols is described here. YouTube uses JavaScript to compute parts of the URLs. youtube-dl executes this JavaScript, just like a web browser.

((WN)) Could you also explain in brief how youtube-dl functions, and how the maintainers had intended it to be used?

((Philipp Hagemeister)) youtube-dl downloads and plays videos and music, just like any other web browser, from over 1000 different services. Its uses are varied: It enables video playback on many devices (e.g. Raspberry Pi) where the video services don’t work properly, it makes high-quality video playable for people with a bad or no Internet connection, it enables disabled users to use tools to play videos, and it is used for archival and research.

((WN)) What do you think of the DMCA notice?

((Philipp Hagemeister)) I think it is not warranted because youtube-dl is entirely legal. As the DMCA notice has no effect for me personally, I’m not really the right person to address it.

((WN)) Why were the copyrighted tests in the source code? Could they be replaced?

((Philipp Hagemeister)) I’m not sure why, but my guess is that users requested support for these videos and thus they were added as test cases. They can be removed trivially, without losing any function of youtube-dl.

((WN)) Are you aware Electronic Frontier Foundation said it was a “disappointing and counterproductive move”? What do you think should be the next steps?

((Philipp Hagemeister)) Yes, and I concur. I’m no longer involved in the project. If I were, I would probably just remove the test cases, block these music videos (RIAA is not worth the trouble for me, that can be done by other projects), and get the project back online.I understand people who think differently.

Tuesday, March 13, 2007

Just two weeks after China sent global markets into a tumble, growing worries about sub-prime mortgage lenders in the United States are sending jitters through stock exchanges around the globe. The three major US stock market indices each fell two percent from Monday’s closing price, marking the second-biggest loss of 2007. London’s FTSE 100, Paris’ CAC 40 and Frankfurt’s DAX 30 indices each closed down more than 1 percent.

As the Mortgage Bankers Association reported that late payments on mortgages and home foreclosures in the US homes rose in the fourth quarter to their highest level in years, investors are concerned that not only US banks but also multi-national banks around the globe could have exposure.

More than two dozen sub-prime lenders have closed or sold operations as defaults on those mortgages have risen. “The delinquencies and defaults have started to soar,” said Nicolas Retsinas, director of Housing Studies at Harvard University. “A lot of these lenders started to make loans and lost track of some of the fundamentals.”

New Century Financial Corporation, the second largest sub-prime lender in the US, has recently revealed that its creditors were no longer providing funds and, further, it has become subject of an SEC investigation. The New York Stock Exchange has said it will delist New Century. Accredited Home Loan Lending, another major sub-prime mortgage specialist, said it was seeking fresh capital and waivers on its lending covenants.

Sub-prime lenders provide mortgages to people who do not qualify for loans from mainstream lenders, typically due to their credit histories. The lenders then bundle these mortgages as collateral for loans that they obtain from other financing firms, such as GMAC. Such firms can then repackage these loans and sell them as mortgage-backed securities. These securities may end up in the hands of major multi-national banks such as Citibank, HSBC, and Commerzbank. Each level of lending assumes a level of financial risk, but trouble can arise when the risk tolerances are exceeded. If too many homeowners default on their mortgages, the sub-prime lender can end up defaulting. If too many of these lenders default, as it is feared may be happening now, the defaults can cascade upward. While no one is predicting major bank failures at this point, it is feared that their profits could be hurt.

Friday, March 10, 2017

UK chancellor Philip Hammond announced his 2017 budget on Wednesday, which included a £2 billion pledge to social care and a tax hike on the self-employed. It was accused of breaking Conservative Party manifesto promises.

It was announced there will be a 2% increase in national insurance contributions for the self-employed, with chancellor Philip Hammond citing worries that people were choosing to become self-employed in order to pay lower taxes and his perception of unfairness in the different rates paid by employees and self-employees. There were accusations this change in policy goes against the manifesto promises the Conservative Party ran on in 2015, which promised four times that there would be no increase in national insurance rates. Conservative MP Anna Soubry tweeted saying she believed these new measures would be unpopular as many would see them as unfair. The leader of the opposition, Jeremy Corbyn, claimed the new measures will not clamp down on people whose self-employment is just for tax benefits, instead causing problems for those legitimately self-employed, arguing that if they are to start paying similar tax rates to the employed then they should get rights such as statutory maternity pay. The think tank Resolution claimed, however, this increase is outweighed by other government policies and is, therefore, a good move.

In addition to this, the chancellor announced a £2 billion pledge to social care over the next three years, saying he was aware of the stress the ageing population is having on the NHS and social care. Liberal Democrat Norman Lamb described the amount as “wholly inadequate”, saying much more is needed to pay for an increase in care demands due to the ageing population. The lowest threshold at which shareholders pay dividend taxes is to be lowered from £5,000 to £2,000 claiming that the taxes for dividends provided “an extremely generous tax break for investors with substantial share portfolios”. Other budget announcements include an additional £325 million for the NHS, £90 million transport spending for the North of England, £20 million to support campaigning against violence against girls and women and a slight increase in funding for the devolved governments.

Saturday, September 18, 2010

United Airlines and Continental Airlines shareholders yesterday approved a US$3.2 billion merger that would create the world’s largest airline. 98% of shareholders approved of the deal.

In the deal, United stockholders will hold 55% of the company, while Continental stockholders will get 1.05 new shares for each Continental share and will hold the remaining 45% of the company. The deal was earlier approved by United States antitrust regulators. The new company, which keeps United’s name but Continental’s globe logo, has little overlap in its route network, and will be headed by Continental CEO Jeff Smisek.

The new company surpasses rival Delta Air Lines, which merged only a year ago, to become the world’s largest.

The airlines expect to complete the merger by October 1 of this year, and customers can expect to see major changes in spring 2011, when the new United will rebrand their kiosks, worker uniforms and airplanes. The new company is calling the day when they rebrand “Customer Day One”.

“Our stockholders recognized the value of bringing together Continental and United to create a platform for increased profitability and sustainable, long-term growth,” Smisek announced after the merger.

Some analysts agree. “No further obstacles to the merger are likely at this point,” Jim Corridore, Standard & Poor’s equity analyst, said to investors. Corridore also said that “we are positive on the planned merger, which we think creates an extremely strong global route network with opportunities for cost and revenue synergies.”

Other people were less positive about the merger. Rick Seaney, CEO of FareCompare.com, said that “losing a major competitor is likely to make prices rise — all things equal on the economy and fuel prices.” Also, a lawsuit in California accusing the merger of monopolizing the market and raising fares has been filed by lawyer Joseph Alioto, who claims that he represents consumers. United and Continental have defended themselves against accusations of monopolizing the airline industry by stating that they compete with low-cost carriers such as Southwest Airlines on three quarters of their US network. The airlines have also said that the low overlap and the low-cost competition will bring fares to heel.

Tuesday, September 20, 2005

US doctors are to interview 12 patients with a view to performing the first ever transplant of a human face.

The Cleveland Clinic will choose between seven women and five men to find the person most suited for the experimental procedure, which is a radical and controversial solution to extreme facial scarring or disfigurement.

Having practiced the procedure on bodies donated for medical research, the Cleveland Clinic team believe they have a 50% chance of success. The procedure will not live up to science-fiction predictions and give the recipient the appearance of the donor; the underlying bone structure is the deciding factor in the final appearance. The new face will end up resembling neither the donor nor recipient.

Surgeons in several other countries have announced being ready to perform this procedure in the past. However, the risk and non life-threatening nature of disfigurement have meant that gaining approval for the groundbreaking surgery has been difficult. Like many other transplant operations, the recipient would be required to take drugs to prevent tissue rejection for the remainder of their life. These drugs can have side effects and carry their own risks involving the patient’s immune system.

Monday, November 2, 2020

The following is the sixth and final edition of a monthly series chronicling the 2020 United States presidential election. It features original material compiled throughout the previous month after an overview of the month’s biggest stories.

This month’s spotlight on the campaign trail: the Free and Equal Elections Foundation holds two presidential debates, three candidates who did not participate in those debates give their final pleas to voters, and three political pundits give their predictions on the outcome of the election.