clustermonkey

OKay, it was ages ago since I actually posted something here but I have had alot to do and all that random crap ;)

Well, I got the system to run! Firstly I could only get it to run with half of the RAM, 16 GB, and that was fine for the purpose I had started it that time.
I only started it and ran it from a USB stick with Tin Hat Gentoo because a friend of mine needed the computational power for a contest he was in (A biological genetik algorithm of some sort) and in the meanwhile I successfully located and bought 4-pin molex to SATA power converters and plugged in the harddisk into the system.

After som fiddling with the RAM, trying more or less every different seating combo..., I managed to get it to accept 32 GB but for some odd reason it can only use 28GB of that (which I found a vague explanation about the 4GB that is "gone" is for PCIe somehow)

But I am installing the server as I write this and man it is a monster. It recompiled the entire system, from a empty tree (emerge -e @world) in roughly 1h 23m, it is insane.

This thursday I went to the DHL Servicepoint and picked up the motherboard finally! The DHL crew ain't the smartest because they hadn't left it off at the servicepoint that is like 1 min away from me, walking distance.
I had to go to the one that was 20 mins away with bus... Oh well now I have the motherboard. (Pictures are coming soon)

I have looked at it and man it looks nice, such a difference from the Supermicro one, the power regulation is quite good, the layout makes sense and it just looks more clean, which is nice.

I will be moving this blog from blogg.se to my own domain whenever I get the time to set it up on my server that hosts my homepage.

And now that I got the mounting holes done correctly, just took me 5 damn tries lol :P, I will start to post alot more than before. Because now I will finally be able to start the system and build the chassi and everything :)

I just recieved the confirmation that they have sent my motherboard to me, so it'll be arriving in like a day or two \o/

I can't wait to test what this hardware is actually capable and seeing as I now have a bit more competent motherboard at my disposal I'll have to think up interesting ways of using it.

For example, I now have 4 Gbit LAN ports on the motherboard instead of 2 which means I can have 4 slave colonies attach to the motherboard without having to extend it with PCIe LAN cards and each slave colony will be able to have up to 23 slaves depending on what switch I decide to use.

The first colony will "only" be able to have 7 slaves because I only have a 8 port switch, but that can easily be changed :D

Plus that I now have a whole bunch of PCI/PCIe ports of different speeds to use, which gives me the oppertunity of extending the master with maybe some cheap graphic card to start fiddling with OpenCL or a SATA/SAS controller card for more disks :)

I'll keep you guys posted on the progress and soon there will be some more pictures again.

I have finally ordered my replacement motherboard. It should arrive within a week or two so I can finally continue with this dormant project lol :P

I'll probably just play around with the system for a week or 3, I have never had such a powerful machine before, but then I am going to get down to business and convert it into a VM host machine, convert my laptop to binpkgs and stuff like that.

Let the game begin soon!

Sidenote: The motherboard I am getting is a Tyan S8230GM4NR, which is a quite more potent motherboard than I had before :)

Well, after a veery long period of not having much time for stuff at all and alot of irl projects taking my time I have managed to get a part-time job while I study so I can soon replace my fried mobo!

The timeframe will probably be like within 2-3 months because I have to get it shipped and such and this also means that I should be more active on my blogg or what do you guys think? ;)

Well, I can hopefully post alot more now than I have done before and perhaps I soon can start to update with real project status, kindof sucks to have a 6 months dead period just because I chose a very bad period to start studying lol xD

After a rather long time of no update here is finally a positive one :D

I have managed to get ahold of a person that will help me get a motherboard from the states to me in Sweden and even with all shipping included I save about 200$, damn what a difference :P

I have been down for roughly a week because I decided to re arrange my partitions and start using LVM, but then I hit a rather strange bug, cryptsetup fails and bam I cannot use my computer without a liveCD lol.
Luckily thanks to blueness and Zorry of #gentoo-hardened @ Freenode I am now up and running in a native environment finally :)

And besides that, FOSDEM is coming up!

Until further updates I wish you all a happy new year and soon I might be able to start to tinker with the server for real :)

Well I have confirmed it now, the motherboard is entirely dead, caput, finito. Maybe I'll make an fancy ornament of it or something but I now have to get a new motherboard...

This means 1. I have to wait and see if I have sufficient funds to be able to get it at the end of this month or next and 2. The waiting for it to arrive to me after I order it.

Ain't life a bitch sometimes? :P

So today I got and fetched the motherboard from the postoffice, it had arrived there during my stay in Germany.
Still a little baffled at the very short description from the support personnel I opened up the motherboard and ye and behold, there were a load of bent CPU pins :/

Well well, CPU pins aren't that hard to put back to where they belong, it only requires a rather good lighting of the work area and something pointy and thin, for me this was a model knife :P

After scanning the sockets and repositioning the pins to where they are supposed to be I decided to do an trial run of the motherboard. I plugged in the PSU, without anything in the board at all, and plugged it into the wall.
The DP2 led never lit up at all, I figured that maybe these server PSUs don't emit power while being in standby so I started it with the help of the motherboard and still no dice, the DP2 refuses to lit :(

Something tells me that I will have to buy a new motherboard :(

I have now gotten the answer from Dustin, apparently in one of the sockets there was bent CPU pins so they are returning the motherboard to me.

Hopefully it isn't broken off so that you could fix the pin and use the motherboard, otherwise I'll just have to buy a new one. They have gone down a bit in price so it is not that big of a hit even tho it means more waiting game :(

Well I have now gotten a confirmal from Dustin that I am allowed to send the motherboard in for RMA so I spent yesterday of demontage and stuff.

Hopefully I'll get the answer rather quick so I know what to do. But I'll probably not see another motherboard this year :/

Plugging in the server PSU yesterday unfortunatly confirmed my fears, the motherboard is dead. The power LED light that is supposed to light up when you connect a powersupply never lit up and I am afraid that the motherboard might be short circuitet.

If it was my EPS12v connector or not I'll leave to the dustin support to judge, I have filed an RMA and see what I get for responce from that.
If it was my connector I will ofc just by a new motherboard, sure it will hurt my wallet but I haven't come this far to just give up lol

And I would also have noticed, because it makes a distinctive sound and smell when burnt electronics are being made :P

I have now after waiting over the weekend for my new PSU to arrive and I just got a text saying it is lying at my local postoffice ready for me to pick it up wuhu!!

Hopefully I'll atleast get a chance to turn it on before I go to bed because today I am going to spend coding at Proxxi.

Oh yeah, if you didn't know, the war of the internet has just begun. Look at this awesome blag post from my friend Etu about it: http://elis.nu/index.php/2010/12/my-day-with-internet-war-part-1/

Well it seems like the motherboard really did not like my own made EPS12v, and I had actually planned that this might happen.
It is after all an very different type of system if you compare what system the PSU usually runs, an single 95-125W CPU, 1-2 HDDs and a 150-225W Graphics card. It was a bit mean trying to run an dual socket system of it and my connector converting skills probably have to be upgraded before I could successfully do such conversion.

So with all that said and done I have now ordered an real 350W server PSU, with 80+ Gold certification and 2 4-pins EPS12v that can be combined into an SSI12v (which seems to be what the server world calls EPS12v 8-pin connectors) and 1 SSI12v 8-pin connector.
So back to the waiting game... atleast they had them in stock so I should have it at the latest by next tuesday

So yesterday just before going to bed I was fiddling with the server and I was going to give it the first go and check if everything was okay.
Well, they aren't... When I managed to get the motherboard started it started to complain with 5 short beeps and 1 long to divide them which according to the information I could find on the net means that the CPU doesn't like life, oh crap I thought.

I swapped the CPUs to see if that could be the error, and of course I double check that everything was fastened as it should be, seated as it should be and not strained in anyway.
Now the motherboard doesn't do anything and I was getting more nervous, I'd hate to brick an motherboard that costs roughly 420€ but then when inspecting the EPS12v connector that I had made I noticed that it was not fastened as it should be so it is is probably that it is not getting power from where it assumes it will and then refuses to start.
I will be repairing the EPS12v connector today and hope for the best, otherwise I'll have to send the motherboard in for repair which would suck because that'd probably take a few weeks.

Atleast here are some pictures of the fully assembled system:

I am now going to switch over to english so that the Gentoo Hardened folks can understand what I am writing :)

I haven't had the time to work any more on the cluster since monday but I have the pictures from monday when I did the secondary EPS12v contact.
I first tried to solder the cables to the pin to make 6-pins into 8-pins but that wasn't such a good idea.
The added thickness that the soldering added to the pin so that the pin didn't even go all the way in so it would not work.

So I took a new pair of cables and stripped them rather far, about 2cm instead of 0.5cm and just used heatshrink to wrap them together with the existing cables. Seeing as I do not have access to an hairdryer or other hot-air blowing source I had to improvise, I set the soldering iron to 350 C and put the heatshrink a few millimeters above the hottest part of the soldering iron and it worked surprisingly well :)



First attempt with soldering the cables together, not that successfull :(



Heatshrink to the rescue! It worked surprisingly well with the heatshrink and the results are heck alot cleaner looking than the soldering ones.



Hopefully I'll be able to test the server today seeing as I haven't had the time to do that right, let's hope that I do not get to smell the awful smell of burnt electronics from my makeshift EPS12v connector frying the PSU :P

Tyckte detta var värt att nämna, USA har tagit ett steg för långt i deras idiotiska kamp mot fildelning och har nu via ett lagförslagt som heter COICA redan! börjat stänga ner torrent sidor.
Trots att lagförslaget bara är ett lagförslag, det är inte inröstat som en lag ännu!
Det är dessutom ett privat företag som står för att stänga ner hemsidor, mer info finns här.

För att rapportera kring lite om klustret, jag ska idag iväg till Proxxi och låna en skärm så att jag kan konfa mastern och faktiskt se vad jag gör lol :P
(Om ni är i stockholms trakten så får ni gärna titta förbi där, det är ett allmänt hak för datorentusiaster och teknik intresserade i närheten av kth,  mer info finns på deras hemsida)
Men hela planen för idag är att skaffa skärm, skaffa AWG18 tråd så att jag kan göra klart den andra EPS12v kontakten och löda ihop den så att jag faktiskt kan starta upp hela schabraket idag förhoppningsvis. Då jag även ska leka mattelärare för en kompis så kanske jag inte kommer så lång som att sätta igång med att konfa klustret, men sätta ihop delarna borde jag lyckas hinna med :)

Idag träffade jag Gnitset från #foss-sthlm@freenode och fick 7st moderkortsdistanser av honom så att jag kan fortsätta  bygget av chassit.
Jag skulle även se till att få den andra EPS12v kontakten gjorde och det var lättare sagt än gjort för då måste man först ta bort de existerande pinnar, vilket inte är det enklaste.

Jag började ta bort pinnarna som man brukar göra när man ska sleeva dem, använde ett par staples från en häftapparat och tryckte in dem i pinnarna från den sidan som ger kontakt med moderkortet, vilket gör att 2st hakar på pinnen trycks in och man kan dra ut pinnen från kontakten.
När jag lyckats ta ut 6 av 8 pinnar på EPS12v kontakten som satt på konvertern så lyckades jag dra loss sladden på de sista 2 -.-
När jag skulle ta bort de 2 så lyckades jag få ett stort jack i tummen som gjorde det ofantligt mycket svårare att ta bort pinnarna från 6-pins PCIe kontakten, men med lite trixande så lyckades jag med det också och valde att sluta där innan jag skadar mig själv mer lol
Något som ska noteras är att pinnarna på PCIe 6-pins kontakten som satt i PSUn satt mycket mycket lösare än de som satt i EPS12v kontakten på konverteringskabeln.

Lite bilder:


Moderkortsdistanser på plats, om ni undrar hur jag gjorde så var det bara att borra ett hål på ca 3-4 mm diameter och skruva i, älskar trä ibland :)



Moderkort på plats som den ska

Edit:
Som ni kanske märker i detta inlägg så valde jag att strippa en PCIe 6-pins istället och det blir inte på en separat 12v rail heller, men det kanske kommer i framtiden, men då blir det att göra om hela PCIe kabeln med AWG 16 tråd istället för att säkra det från överbelastning

Behöver införskaffa lite små saker som krympslang och moderkortsdistanser och såg att de hade fina saker hos Kjell och Co.
En box med ca 90st krympslangar av diverse tjocklekar och pc skruv set var siktet inställd på och när jag väl kom till butik så tog de fram krympslangen och sen när de skulle ta fram pc skruv settet så fanns enbart 2st distanser per påse... No way att jag betalar 49kr för 2 st idiotiska distanser!

Jag kom hem med en box krympslangar för 99kr dock, blir att leta moderkortsdistanser från vänner och familj helt enkelt :)

När jag började kolla på PSUn så märkte jag en grej, visserligen hade den både en EPS12v kontakt och en P4 kontakt som jag hittat konverterar till EPS12v för men de satt på samma kabel :(
Dvs att all ström skulle gå igenom samma stackars kablar och då man använder AWG 18 kablar för PSUn så klarar de bara av 2.3A st, och då en EPS12v kontakt kan max dra 7A så blir det väldig överbelastning på dessa tunna kablar om man kör 2st EPS12v kontakter
(1 tråd klarar max 2.3A och det är 4 kablar per EPS12v kontakt som bär strömmen, dvs 2.3A x 4 klarar en EPS12v:as trådar vilket blir 9.2A totalt och har goda marginaler till de 7A som en EPS12v:a mest ska kunna dra)
Så efter lite snack och diskussion i irc-kanalen #foss-sthlm@freenode så bestämde jag mig för att jag använder en separat 8-pins PCIe kontakt och använder dens 3 12v för att ansluta till de 4st som en EPS12v kontakt behöver och ansluter de 5 jord kablarna till de 4st jord som en EPS12v:a ska ha.
Då ligger denna kontakt dessutom på en annan 12v rail vilket minskar risk för överbelastning av 12v ännu mer

Alla boxar "små", äntligen har alla kommit



Stor box, litet innehåll. Låter som titeln på en ganska dålig kampsports film :)



Här är all hårdvara som kommer utgöra min master-node i klusterprojektet.
Totalt värde: ca 21 000kr, jag har drömt om att kunna göra ett sånt här kluster i snart 3 år, och lika länge har jag skissat på hur det ska se ut och äntligen så händer det.



Shineh moderkort, jag har aldrig förut fått se ett dual socket moderkort. Aldrig ens på långt håll har jag lyckats hitta det, men de är riktigt snygga :)



Powered by Cougar! Man fick med lite smått och gott med PSUn



PSUn är riktigt riktigt snygg, blank metal finish, kablar sleevade med matchande färger och en rejält tilltagen fläkt. Att den har en effektivitet på minst 87%-90% beroende på last är bara pricken på i:et, det ska bli ett nöje att stifta bekantskap med detta PSU

Nu börjar saker komma på plats och eftersom jag kommer bli mördad ett par gånger om jag inte skriver någon form av logg över mitt bygge/forskningsprojekt så är det väl bäst att man börjar.

Hårdvara börjar trilla in, de flesta sakerna fick jag för nästan en månad sedan så som diskar, fläktar, PSU och RAM och idag ska jag dra och hämta moderkort, CPUerna och kylflänsarna.
För de som undrar så blev det till slut detta som jag kommer ha i master noden:
AMD Opteron 6128 x2
Supermicro 2U passiv kylfläns socket G34 x2
Supermicro moderkort H8DGU-O
Cougar S Power 550W
Western Digital Cavial Black 640GB 32MB Cache x2 (Ska sättas i RAID 0 för ännu mer jummeh performance)
Kingston DDR3 1x4GB 1333 MHz ECC REG CL9 x4

Den switch jag kommer använda för att ansluta slavnoderna till mastern är en HP Procurve 1410-8G vilket är en 8 portars Gigabit switch och som jag, och andra, tror är bara en relabelled 3com switch men det verkar vara bra hårdvara.
Jag tycker om de 3com switcharna jag har fått pilla med förut och även procurves är roliga att ha att göra med så det blir nog bra vilket som.

Sen har jag köpt diverse kruft för att kunna styra servern som ett tangentbord och mus samt skärm :P
Bilder och sånt kommer senare
Edit: Det ska vara 8st 4 GB RDIMM:ar inte 4, glömde att jag köpt till ytterligare 4st från de originala 4 som redan var införskaffade