Hacker Newsnew | past | comments | ask | show | jobs | submit | TacticalCoder's commentslogin

The USB stick hints at a big problem in our trade though: how do you "reboot" your IT infrastructure if it literally burns to the ground? I'm not talking about Google-scale systems (which still couldn't restart from scratch IIUC but they're actually working on it?) but only about SMEs.

How does a medium-sized SME were all the payrolls depends on Sara and her USB stick do if, literally, their servers do catch fire.

You've got backups, then what? How automated is the reinstallation of your typical SME's infra?

The closest I saw to that scenario was some documentary where some little trading firm had just time to fetch the backup hard drives before leaving the building on fire after a plane crashed into it on 9/11. The CEO (I think it was the CEO) was explaining that had he not grabbed a HDD with the backups, the company was done (not that I advice onsite/offline backups on HDDs that you must not forget to grab when the shit hits the fan as a solution btw).

I understand the "just drink the cloud kool-aid" angle: but are SMEs typically doing that?

How many SMEs out there are depending on Sara's knowledge of the USB memory stick and how to use it?

I've definitely seen similar things. And I'm sure many of you did too.

Many houses of cards?


When I took charge of solving backups for the single important box with unique, irreplaceable data -- the accounting system -- at an SME a long time ago, I think I approached it with the right amount of correctness. Therein, losing a day or three of recent data would have been recoverable; losing all of it would have been catastrophic.

I devised a system to perform bare-metal backups onto an easily-swapped, external 2.5" hard drive, using Acronis. I provided a plurality of these hard drives, and they were to be rotated off-site. The system was tolerant of human error and would proceed with making valid, current backups even if the drives were rotated incorrectly, or if not rotated at all on any given day. The backup drives each had complete file history (yay shadow copies) from an ever-advancing date, so any given drive could be used as a time machine of varying resolution, and also as the single source from which to independently start fresh.

I'd watch the logs to see that it was done, and for the most part: Whoever was assigned to that role normally did it properly-enough.

I documented it and showed the other technical folks how it works.

Sometimes I'd wander back and make sure the backup drives weren't accumulating on-site (there should never be more than 2 on-site). I'd periodically test these backups by restoring them completely onto identical hardware, to make sure the system hadn't got crufted up somehow and that it still continued to perform its task of restoring a working system from zero.

It worked fine for years and years. We never had to use that backup, but I had every confidence that it would be useful if that ever became necessary.

Eventually, my role changed and those things rather officially became Not My Problem.

Later, they moved the accounting system from that lineage of stout Proliant boxes to a trash-tier small-form 1u Lenovo machine that someone found used, on eBay, for cheap.

Backups are handled by the clown, somehow. The last I heard anything about it, the person doing the talking was very pleased with the money they'd saved and that they'd no longer have to pay "extortion" to Acronis.

I have every expectation that nobody has ever restored these backups. They're probably relying on the sheer hope that they'll never have to restore them, much less from zero.

And I also hope they never have to restore them, lest they may find out exactly what that data is worth to them.


> How does a medium-sized SME were all the payrolls depends on Sara and her USB stick do if, literally, their servers do catch fire.

Like every job, we overestimate our importance.

What do they do? They pay everyone the same as last month as a temporary measure, ask you to talk to your manager if your pay should be more this month, warn everyone that they're going to recalculate the payroll and adjust any differences next month. Then they calculate everyone's pay from the inputs, which really isn't such a hard problem when the alternative is failure. Maybe they pay some fancy consultants or an SAAS provider for a few months. Maybe they have to cut a few corners. Maybe they even get fined by their state's DoL. Life goes on.


> How many SMEs out there are depending on Sara's knowledge of the USB memory stick and how to use it?

I think at least in part, that is the point: orgs are missing the part of the equation where the institutional and organizational knowledge is critical. Sure, the code to accomplish parts B and C can be re-duct-taped together in a month or so by off-shore, or maybe an agent... but part A, its plumbing, and why it does what it does the way it does it due to historical failures and the knowledge behind that is probably what keeps it going.

Those things are learned starting at the ground level by bumping into them in the trenches.


The company just shuts down and its customers switch to competitors. This is economically efficient. The redundancy of a company is another company. It's a bit like how we don't insist on every server running two CPUs in lockstep in case one fails, because we have more than one server to handle requests.

>The USB stick hints at a big problem in our trade though: how do you "reboot" your IT infrastructure if it literally burns to the ground? I'm not talking about Google-scale systems (which still couldn't restart from scratch IIUC but they're actually working on it?) but only about SMEs.

Maersk ground to a halt because it got done nearly 100% by cryptolocker. IIRC they went to hard copy records, called everyone, got all of IT together with some company credit cards to get new laptops and flash drives and shit and literally rebuilt their infra from scratch.

https://www.itnews.com.au/news/maersk-had-to-reinstall-all-i...

I read a better post mortem but thats the highlights.

>How many SMEs out there are depending on Sara's knowledge of the USB memory stick and how to use it?

Part of my day job is finding, documenting and remediating these sort of issues.

"The CEO Coded this application in VB5 15 years ago, the entire business relies on it, theres no source code, theres no binary backups and the one computer it runs on just had its PSU fail"

"Theres a cron somewhere that compresses, zips and transports the payroll database interstate, outside of our network, before our weekly pay run"

"Theres been no documentation of this environment for 20 years, most of the hardware is that old, and the team that developed it just sold all their shares and left"

This shit is my life lmao.

Theres obviously some bias, because the good companies aren't asking me to do it for them. But I make a decent living examining, documenting and remediating this shit.


How did you get into that line of work? Sounds really interesting.

Refusal to pick a silo, having a knack for troubleshooting, falling into consulting. It just sort of happened. Helps to be extremely jaded too. My kneekjerk disbelief that something is good, documented or even functional makes me well suited to taking over new clients and finding where all their bodies are buried.

One of my favorite jobs early in my career was working for a really shonky wireless isp. The majority of the network was built by sales people using terrible tools with no documentation. I actually cant overstate how bad they were originally, they had entire areas of network with no recorded network config or credentials. My daily workflow was getting a ticket from a customer I had never heard of > trying to figure out where they were and what services they had (2 of their 3 billing systems were offline, and I often had to grep out information from a sqldump to find this stuff) > performing a discovery, L2 upwards of their infrastructure > semi offensively trying to authenticate into their infrastructure > resolve and document so that other people can reliably service them. All while pretending this was absolutely normal to the customer. Turns out there were lots of ISPs in the same boat, and turns out there's lots of non isp businesses in the same boat.


> How does a medium-sized SME were all the payrolls depends on Sara and her USB stick do if, literally, their servers do catch fire.

The SpecOps guys have the following bit of wisdom on offer: "Two is one and one is none".


And a backup you haven't verified you can restore from isn't one.

> In Europe governments make up such a big part of GDP now in some cases nearly half of it ...

Oh more than half. In France it's 59% officially. And then there are the fake "private" companies that are actually owned by french-state apparatchiks and operating like the various state monopolies (like utility companies): so the real number is higher than 59%. France has probably more than 2/3rd of its GDP that is public spending. It's basically a planned economy.

A planned economy with the only expected result of a planned economy: the public debt of France is 115% of the GDP growing. Inflation is through the roof (you think gas prices are high in the US?). And they have zero clue as to how they're going to pay their empty promises of pensions to the aging population.

But what's really amazing in a country like France where 2/3rd of the GDP is public spending is this: publications constantly hammer the exact same message as in TFA: "We should tax the rich!". The french Piketty (who's btw never worked for a second in the private sector in his life: a pure product from the socialist french education system who's exceptionally good at creating state-lovers ever begging for more taxes) is mentioned in TFA.

2/3rd of the GDP being public spendings but instead of trying to get out of that planned economy the message hammered by all the media (who are either owned by the media of by the french-state apparatchiks) is: "Tax the rich".

The delicious irony of the 3 first of the only 5 companies France has in the Top 100 by market cap (and none in the Top 50) being three companies selling luxury goods and bringing money into France by selling luxury products outside of France is of course not lost on people.

There's LVMH, L'Oreal and Hermes exporting like mad luxury products and bringing in money from overseas into France and, instead of giving the people owning these companies medals, France explains that you should "tax the rich".

Yeah. But no. I just don't buy it.

I'd rather be poor in a capitalistic society than live as a slave in a planned communist economy.

"Better dead than red" FWIW too.


You'd rather be poor in the USA than a slave in... modern France?

I'm flagging because HN guidelines asks to keep political content to a minimum. This one of those classic "evil rich" / "gentle poor with holes in his sofa" communo-marxist rant.

Out of HN.

I'm here for tech news. Not for pro-libertarian posts. Not for pro-right posts. Not for pro-left posts.

I flag any political content (no matter the aisle) and you should too.


> Yeah it's only the third largest economy in the world

You can both be the 3rd biggest economy in the world and still only be 1/10th of US+China GDPs combined.

And only three companies in the Top 100 for Germany:

https://companiesmarketcap.com/

Germany is the kingdom of the "mittelstand": many, many, many SMEs.

Both GP and you are right: it's the 3rd largest economy in the world and yet it's simply not that big.

https://en.wikipedia.org/wiki/Mittelstand

In other words: I expect this German DNS SNAFU to have 0.000000001% impact on the world's GDP this year.


> In other words: I expect this German DNS SNAFU to have 0.000000001% impact on the world's GDP this year.

126 trillion USD * 0.00000000001 = 1260USD

I'm pretty sure the impact was higher than that ;)


How is 1/10th the size of number 1 and 2 COMBINED small? In what world is that a small number? Especially as those two are 1.8 billion people vs 0.08 billion for Germany

This comparison threw me for such a loop. What an odd way to present a point.

what's SME?

Small/Medium enterprises

> But doing an explicit SHA compare - that's just...not something I would've ever thought of. Wild.

If I'm not mistaken SGI (Silicon Graphics, Inc.) was already doing that to prevent regression 40 years ago: maybe not SHA but they were taking "screenshots" of the entire screen at a time t and some kind of checksum to then verify (without having to compare every single pixel in the happy case) that enhancement/optimization to their rendering pipeline not supposed to change the output indeed did indeed generate the exact same image as before.

It's basically a 40 years old technique: not too sure what's that wild about it.


Sure, it's been done before, and I'm sure not just limited to SGI, but no one does this for regular apps these days - never heard of it before. I just find it neat that Codex came up with this - not something I ever would have.

Edit: I'm not saying no one does checksums to compare files (lol). I'm saying no one takes screenshots at specific timestamps within an app or game's lifecycle and then compares them to ensure they're identical.

Edit 2: Whoops, looks like I'm wrong and this is apparently a pretty common thing (but not at the startups I've worked at, /shrug). I still think it's cool that Codex is doing it without being told to, though.


> but no one does this for regular apps these days - never heard of it before

Everyone does this to match files as identical, be it sha, md5, or something else. I cannot imagine any other method such that it would first come to mind easily you would be doing to check if two files are the same.

I don't mean to offend but I quite literally mean everyone does this. Every software updater, game patcher, checking if two binary files are identical (pixel perfect/lossless in this case: BMP, PNG created by same encoder off same inputs would qualify, JPG would likely not), all of them do exactly this.

GPT-Analysis or a similarity and image chunk hashing would not be the first thing you turn to if what you wanted was exact identical pixel perfect. I am curious what your background is if this is the case.


Not sure if you're getting what I'm saying.

No one that I've seen takes automated screenshots of webapps or games or what have you at pre-determined timestamps to make sure the app looks pixel-identical with every change.

(regardless of the method; the SHA'ing isn't the point here, the point is that it's a shortcut instead of "inspect the image for any regressions", since we don't need to inspect the image at all if it is identical)


> No one takes automated screenshots of webapps or games or what have you at pre-determined timestamps to make sure the app looks identical with every change.

I'm confused. We have done this at every place I have ever worked, it's very standard. Set timestamps, post-action, pre-action & on dozens to hundreds of combinations of OS and rendering engines. This includes pre LLM, using similarity and perceptual hashing, screenshot-ing single DOM elements during hover and off hover, both fuzzy and pixel perfect.


Huh! Well, I stand corrected. I've never seen that done (but I've only worked at startups with < 20 headcount for my entire software career so far, so that might be why).

Huh. Were they anywhere that pixel perfection was necessary such as games, or required constant browser universal testing for compliance, accessibility, being required to support cross platform?

Have any of your places used a service such as Saucelabs or Browserstack or rolled their own similar inhouse, or seen such as https://percy.io/how-it-works (random example; not affiliated or recommending this)?

I am hope I was not being too rude about it, not my intent, mostly surprising to me because a service like Browserstack is a decade and a half old already and the concept predates that.


I was wrong & you called me out on it, not rude, all good.

My first software job out of college was actually a QA Automation / SDET position, wrote an automated framework with Ruby + Selenium + Browserstack which did take screenshots of the app, but the app loaded dynamic content and there were frequent feature adjustments so no two screenshots were ever identical.

All other jobs I've had since then have been writing smart contracts for Ethereum apps - 100% backend, (I hate having to deal with frontend) so all our tests were just units & coverage & what have you.

I suppose if your environment holds constant and your features don't change frontend structure or behavior (eg refactors), then this is what you should expect.

Though, do note that this only works because my app is based on a tick/game-loop system without callbacks; if this was the standard game-development pattern of callbacks & message handling (especially w/ React / JS) to invoke events, it wouldn't work, because the timing would be slightly different each time, and an enemy would be a few pixels to the left/right of its position in the past run.


Obligatory https://m.xkcd.com/1053/ reference, but you're taking this in good stride and that's excellent. :)

If you want to go further down this direction there are all kinds of cool things you can do. There are ways to like XOR bitmaps so pixels which aren't identical show up as white and the rest are black, and the like; if you're working with something else you can look into perceptual hashing although that's a lot more computationally expensive.

Oh! And edge detection! Canny edge detectors are cheap and deterministic and wonderful for all manner of this storm.


Oh yeah, I did a deep-dive into neural networks (both artificial and human) for vision processing, it's super dope stuff. The human vision processing system is remarkably similar to some of the AI stuff we've built for image processing!


...yes, I'm aware of what a checksum is.

> For those in the finance space, are you actually seeing any real AI tools being used? Like for actual operational tasks?

> I've really only seen it used for research / exploration thus far

Summaries and translation for sure.

Speaking with devs in the field I know that AI tools are used to summarize and extract data from... PDFs. Now, thankfully, LLMs got better at answering "How many 'r' in 'strawberry" and it looks like they're good enough for summarizing PDFs and extracting key numbers but I'd still be cautious.

And I've got a friend who's a translator specifically for financial documents: she's a contractor and getting about 1/10th of the work (and 1/10th of the pay) she used to have for now she's only tasked to verify that the translations are correct. Of course she already had lots of tools, way before he LLM era, automating some of her work but she was still billing he use of those tools. Now LLMs are doing nearly all the work and not "for her": it's happening upstream and she only gets the output of the LLMs and has to verify them. And there aren't that many errors.


> Rather than relocate, we can make discussion of climate change illegal or just tax the blue states to build a sea wall around the entire city

Like in The Netherlands?

https://en.wikipedia.org/wiki/Delta_Works


There's a museum in New Orleans that has a Katrina display and it turns out that they did indeed call in Dutch experts to advise them. The Dutch gave them sensible ideas like building low elevation parks that could flood without issue and hold lots of water, instead of concrete spillways and drainage that just moves water fast until it fails catastrophically when inundated. Louisiana being Louisiana, it was all ignored.

The museum convinced me that New Orleans is doomed in so many ways. Everything from the Atchafalaya ORCS to the paving over of wetlands to build the city to the destruction of the Plaquemines marsh lands to the southeast of the city all seem to be maximally unhelpful for preventing storm damage.


The reality is that New Orleans is simply not important enough.

Even the biggest ultra conservative GOP voting redneck will have to admit that America can't survive without NYC which is why it will get it's seawall.


This is very fascinating from a cultural viewpoint. Some cities in Europe are important just for the history and not economy like Venice, or lesser extent Rome. When the Russian attack started, people moaned about the old buildings and the culture in Kyiv. (Ofcourse the attack itself was inmoral).

What I get is New Orleans has an unique culture and history. Most people in the US dont think it is worth to preserve?


When you look at who the rest of Louisiana voted for, they don't even want to preserve New Orleans. They're literally terrified of it and were elected on the promise to subjugate it.

What they really want to preserve is stuff like this: https://www.youtube.com/live/mHljI5JbnTM?si=dReL9sZKiqtNlvpr...


Most people in the US think everybody should have access to basic healthcare but we haven't been able to make that happen.

Something like saving New Orleans probably doesn't stand a chance.


I mean this is a sign of the flawed political process. But even in a working democracy if people arent interested, politican mostly wont care. So Americans dont think New Orleans is worth to preserve?

I've been reading more about New Orleans situation this morning and my thinking is changing. It would be nice if we could preserve it, but I didn't really understand how bad the situation is. I don't think it's possible and spending should be focused on relocating people from the area.

New Orleans is probably going to be a fairly small island 20 miles offshore that gets drowned by hurricanes every few years.


It's really getting a lot of upvotes so it's nearly as if people were feeling locked-in and wanted a way out but...

Why would you keep using CC CLI if you want to use the much cheaper DeepSeek v4 models (Flash and Pro): isn't it the opportunity to kiss CC CLI goodbye and use something not controlled by Anthropic?

Anyone here successfully moved from CC CLI to a fully open-source project? I'm asking this as a Claude Code CLI (Sonnet/Opus) user. My "stack" is all open-source: from Linux to Emacs to what-have-you. I'd rather also have open-weight models and a fully open-source (not controlled by a single company) AI CLI.

Any suggestion for something that works well? (by "well" I mean "as well as Claude Code CLI", which is not a panacea so my bar ain't the end of the world either).


CC can use the Flash model for sub-agents and other tasks but keep using the Pro model for the main thread and thinking there. It also has a decent permission system and commands that people are quite used to.

Personally I’d say that the closest alternative you’d get is OpenCode.

They are TUI focused and also do have a web/desktop version, though that’s not quite as good as Claude’s Desktop app (and Claude Code functionality within it) yet. The TUI itself is solid though and they support a lot of different providers and models (/connect and /models).

For GUI driven stuff, I’ve personally had good experiences with KiloCode (in Visual Studio Code, personally I also liked RooCode of which KiloCode is a fork, but that stopped development) or just using the Zed editor with their built in agent.


Here's Porsche really at its best, on the Nurburgring's Nordschleiffe in 2018, which is arguably the most complicated and diverse track on the planet doing a cool 5m19s (in 2024 a Mercedes AMG GT One was a full 1min10s slower, for example):

https://youtu.be/PQmSUHhP3ug

Any car lover who doesn't know that vid can safely watch it.


He's also king of the "I'll criticize the west but I'll turn a blind-eye to non-democratic countries' wrongdoings". A trait shared with virtually all intellectuals and artists in the west.

There are fights worth fighting: for example there are 300 million women alive who have undergone forced genital mutilation. 300 million ain't cheap change. There are also hundreds of millions of people who applauded the killing of 1200 young civilians who were enjoying life at a music festival "because it's resistance".

Applauding the killing of young unarmed civilians, genitally mutilating women and turning a blind-eye to a regime slaughtering 30 000+ of its own unarmed civilians is where I personally draw the line and consider there are maybe more important things to complain about than, say, "the patriarchal western society built by heterosexual white men" or some other woke non-sense like that.

Now to be honest Banksy did art criticizing war overall, not just war started by the west. So a generous reading could consider that he also criticizes things like the 800 000 deaths during the Hutu vs Tutsi war.

But still overall: lots of balls from western artists when it's about criticizing the west, but tiny tiny nuts when it's about, say, attacking the ideology that is responsible for 300 people enjoying music at the Bataclan and then getting slaughtered.

But these people can live with their own conscience: I speak up and I've got mine.


That's a lot of imaginary flaws in imaginary people, with imaginary numbers as scaffolding.

The moral posture you're criticising is not actually a thing, I personally don't know of any Western intellectual who criticises the West but is fine with FGM for example. But it seems that the fault you find in them is that when they criticise the West, for example, they don't also add a list of grievances against all the other countries (but surely they'd have to speak for 10 hours every time they open their mouths?).

It's also funny how you take the 30,000 Iranian civilians killed at face value, but don't talk about the wrongs of the British empire. And you didn't even mention North Korea once. You see the issue with your reqs?


Are you making art to fill that perceived gap, or just lodging your objection to people doing their own thing? No artist owes you a curriculum of your design.

The Iran problem is a good example: it was wrong of them to massacre civilians, but you cannot fix this by .. bombing more civilians.

So how do you fix a situation, where one party relentlessly attacks all the time? Israel, does what ukraine does- a strip of death around the country- getting wider as the technology to attack it matures.

What do you want the artists to do about it? Part of art's power is shining a light on something we don't notice day to day. Most westeners are against mutilation, what would the art say?

Art will always be about speaking truth to power, and that power will usually be the one closest felt. There's not much value in a swede speaking truth to Nigerian warlords.


> But these people can live with their own conscience: I speak up and I've got mine.

Not sure there's much conscience in Banksy making anti-national chauvinist memes whilst not identifying as any sort of nationalist, but there's even less in dismissing all criticisms of one's own society's treatment of, say, women because some other societies treat them worse.

For all that I don't think posturing graffiti artists are the saviours of humanity, it's difficult not to notice that the groups that actually are tackling FGM are practising Muslims and super-liberal NGOs (in that order) and that the people who raise it to deflect from criticisms of their own society are not represented at all in those efforts. Or are actively campaigning to get women's escape routes from those countries shut down.

Can't really lecture others on losing their sense of perspective about the magnitude of injustices either when a week ago you were expressing outrage at checks post history creatives depicting certain characters in LOTR as non-white!?!


He's a Brit mostly putting art in Britain and so it's naturally that way focused. I've no info what his views would be on forced genital mutilation - probably against but not his area of art like most people.

There's a lot wrong with the world, but it seems not unreasonable for people to more strongly critique things 1) they feel they have some responsibility for or 2) that directly impact them or 3) where their criticisms are more likely to result in positive change.

Oh yes the classic problem of 'the west' always bettering themselves. If they would actually start focusing on the rest of the world, maybe the world would be a wonderful place. Right?

Or maybe, we should look at the problems in our society and try to make it better, instead of just shouting into the void about things we, as nations, can't and wouldn't be and perhaps, shouldn't able to change?


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: