| abstract
| - Town Hall with Ian, Daniel, Cyn and Jeska Linden 10/31/2005 Topic: Estates Daniel Linden: Buenas dias! Daniel Linden: You've arrived early -- wanted to get a good seat? Daniel Linden: That's the first test.... Cyn Linden: Sure Cyn Linden: Hi Jes Cyn Linden: Nice devil Cyn Linden: Yep Cyn Linden: Hi Alliez Jeska Linden: Hey everyone Daniel Linden: Thanks! Jeska Linden: We'll start in just a few more minutes :) Jeska Linden: Hello everyone! Welcome to the Technical Town Hall with Ian Linden, director of operations. Thanks for coming out and Happy Halloween! :) Jeska Linden: First some guidelines for the discussion. We will be focusing this Town Hall on questions about technical issues, please don't submit off-topic questions, as they will be skipped over. Jeska Linden: Ian is going to open with a brief introduction and then we will go into an open QA. Questions can be sent to me (Jeska Linden) in the form of an IM. Ian Linden: I am? Jeska Linden: Also, for those who have joined us here, I'd like to ask you to please hold down the chat and offers of friendship until after the meeting as it can be distracting. Jeska Linden: Ian, take it away. Ian Linden: uh.. all you concerns will be fixed in, uh, 1.8, I, uh, promise Ian Linden: ;) Ian Linden: actaully, since I know there are alot of performance questions Ian Linden: I think everyone should read the docs we've posted Ian Linden: link Ian Linden: Ian Linden: not very in-depth as yet but they'll grow Ian Linden: and it's a good baseline Ian Linden: err, that's all I've got, I'd rather just spend the time on questions Jeska Linden: Excellent, send them on over to me in IM :) Jeska Linden: Nexus Nash: What's the status on estate tools v2 ? (more stats (scripts, objects etc), more control to deed, etc) Ian Linden: forthcoming, but not immediately Ian Linden: I can't give a date, unfortunately Ian Linden: I think really these features will come in several small bits Ian Linden: instead of one big overhaul Ian Linden: we're focusing right now on fixing the existing bugs Ian Linden: and lowering the crash rate Ian Linden: after that I think we'll be able to start improving those tools Ian Linden: it's clear that they are key in the long term Ian Linden: sorry if that sounds like a cop-out, the schedule just isn't firm Ian Linden: next q? Jeska Linden: Surreal Farber: Since a private sim represents substantial investment, when will be able to register with LL things such as survivorship and/or designated representative. Ian Linden: good question, I have no idea Daniel Linden: I'll take it! Ian Linden: daniel? Daniel Linden: We're working on plans as we speak to give you all a MUCH higher level of control... Daniel Linden: ...administratively, over your estates. Daniel Linden: Cyn and I will be making the rounds to personally speak with you about these plans... Daniel Linden: ...and make sure we moving in the direction you want. Ian Linden: another "coming soon" I guess Daniel Linden: Once we've finalized the details...we'll move pretty quickly. Daniel Linden: Stay tuned! Jeska Linden: Cutter Rubio: What provisions, if any, is LL making for keeping the private sims up to date in terms of hardware performance? Ian Linden: that I can answer Ian Linden: right now we replace old machines on a rough schedule Ian Linden: the basic plan is three years although we've been shorter than that so far Ian Linden: ultimately we won't keep class 2 hardware in service forever Ian Linden: if you're on a class that gets upgraded/replaced Ian Linden: then you'll be moved up the latest stuff Ian Linden: this makes the most sense for us from a deployment perspective Bub Linden: Script run-time error Bub Linden: Too Many Listens Ian Linden: I can't give a timeline for the death of class 2, but hopefully it will be sooner than the 3-year anniversary for those machines Ian Linden: we've considered offering a way to pay to move up ahead of schedule Ian Linden: but it doesn't seem like that would make alot of sense Ian Linden: given the high cost and relatively modest performance gains Ian Linden: but feel free to disagree with that ;) Ian Linden: next? Jeska Linden: David Valentino: With the new and more accurate script count/readings, is there a target for max number of scripts we should be looking at for optimal sim performance? Say in a 75% prim usage sim? Ian Linden: hmm Ian Linden: unfortunately that depends more on what the scripts are doing than how many there are Ian Linden: clearly whenever we get above 2500 or so things start to slow down Ian Linden: but nominally when you run a ton of scripts all you'll do is slow down the other scripts Ian Linden: not the sim Ian Linden: that's the goal anyway Ian Linden: it's just the number for the sim Ian Linden: the old number, it turns out, failed to count scripts on link children Ian Linden: oops Ian Linden: so they tell me Ian Linden: so, clearly we need better accounting/scheduling Ian Linden: as mentioned in the first question Ian Linden: next? Jeska Linden: Nexus Nash: Any reason why TD gets hammered post 1.7 whenever there is the slightless load on the server? Ian Linden: yeah I'd like to know too Ian Linden: a Ian Linden: actually, we're not running havok in lockstep w/ the rest of the sim Ian Linden: nominally this is more efficient Ian Linden: but if the sim drops below 45 fps then havok suffers as well Ian Linden: and vice versa Ian Linden: I think we're going to find there are efficiency bugs in the current code Ian Linden: yeah, it's not as good as it should be Ian Linden: I would say, give us some time to patch this release Ian Linden: paulie: I'd like to put that Q in the queue, jeska ok? Jeska Linden: Sure Jeska Linden: Please send questions in IM form to me :) Ian Linden: anway, we've done some grid-wide sim stats analysis Ian Linden: and I think we're zeroing in on some problems Ian Linden: I'd have to go offline to get more detail Ian Linden: next q? Jeska Linden: Mike Getting: Do you allocate faster servers to islands with a higher average load, or do all islands get whatever's current when they're first created? Ian Linden: no, they get what they got when they were created Ian Linden: this seems to be the most fair Ian Linden: esp. since we can't move them between servers while they're running Ian Linden: yet Ian Linden: next q? Jeska Linden: Anshe Chung: In Dreamland we have more than 300 people who own land deeds. People buy and sell those every day, but the process is manual due to lack of tools.... Jeska Linden: When will you recognize this existing market and provide tools that allow us to automate the trading ofdeeds in zoned sims? Ian Linden: daniel, maybe you can comment on that? Daniel Linden: Yes! Daniel Linden: Anshe, this issue is in our plans for THIS quarter. Jeska Linden: When will you recognize this existing market and provide tools that allow us to automate the trading ofdeeds in zoned sims? Jeska Linden: opps Ian Linden: I think this sort of thing is a clear goal for us Daniel Linden: We plan to start creating and rollilng out these tools (and more) as they're complete. Ian Linden: the mainland can't be the only estate where land deeds are automated Ian Linden: again, look for these in a series of small steps Ian Linden: I think the "one giant release" model has been shown to have some problems ;) Jeska Linden: Mike Getting: How do we find out what class we're in? Ian Linden: yeah, that'd be handy, eh Ian Linden: tell you what, after this is over, I'll post that to the sim stats wiki page Daniel Linden: We're going to solidify these type of arrangements and put the tools into Second Life -- it'll be better for owners, renters, vistors....this is a big goal for us NOW. Ian Linden: right now you have to look at the hostname of the server in the About SL window Ian Linden: I suppose it would be a good idea to report it directly Ian Linden: next? Jeska Linden: paulie Femto: I know LL claims no performance degradation between sims on the same server, but can we find out what estates we are stacked with so that we could arrange large events so as not to conflict with stacked neihgbors events? Ian Linden: interesting Ian Linden: you can find out, again, by matching the hostname in About SL Ian Linden: that tells you, at least, if two are on the same machine Ian Linden: although it's a bit of a pain Ian Linden: yeah, but right now that's pretty much all you've got, unfortunately Ian Linden: the number of sims stacked on a machine is generally fixed for each class Ian Linden: at least Ian Linden: anyway, sorry I can't offer a better method Ian Linden: I will say, however, that generaly speaking you shouldn't find persistent issues w/ stacked sims Ian Linden: keeping in mind that ALL class 3s are stacked and everyone knows they're a bit faster than class 2s Ian Linden: which are NEVER stacked Ian Linden: however, we do run into trouble with the memory usage sometimes Ian Linden: and that is a bug Ian Linden: which we will very much try to fix Ian Linden: which should reduce the problems that do occur Ian Linden: next q? Jeska Linden: Wayfinder Wishbringer: Of similar interest... for months we have asked for multiple-manager access of estate tools in private sims. I LL recognizing this need and planning to provide such? Ian Linden: yes, see the first question Ian Linden: I don't think the form these tools will take are yet set in stone Ian Linden: so we'd love to hear your wishes Ian Linden: next q?> Jeska Linden: Vx Shaw: Regarding performance, is there a benefit to replicating the same script, as opposed to similar scripts performing similar functions - for example, if an automated vendor was standardized within my sim? Ian Linden: not really Ian Linden: however, inasmuch as vendors are standardize, if you can update them in an automated way Ian Linden: there are alot of benefits to that Ian Linden: especially if you are able to optimize the script, etc Ian Linden: but no implicit benefit to uniformity Ian Linden: next? Jeska Linden: Cutter Rubio: Right now, if we want a new height map (RAW file) applied to the sim, LL has to do it - any plans to allow estate owners to do that themselves? Would this be one of the future tool possibilities? Ian Linden: daniel, got anything on that one? Ian Linden: I believe that this is a step we'll *have* to take eventually Ian Linden: manual processes just don't work as we scale out Ian Linden: but I'm not sure if that's near or long term Daniel Linden: We want to automated as many steps of the Estate purchase process as possible... Daniel Linden: ....and uploading land form is also something our internal content team wants to make easier... Daniel Linden: ...at this point, our internal tools aren't good enough to allow what you're asking, but everyone wants to ease this part of the process! Ian Linden: I can certainly imagine uploading a raw file to a sim and having it load it in on the fly Ian Linden: but imagination and reality take a while to get together Ian Linden: next? Jeska Linden: cua Curie: What is the current timeline for SL upgrades? What is the target timeframe for adding HTML on a prim? Havok2? mono and the new rendering system? Cyn Linden: Sure Ian Linden: woah Ian Linden: havok2 is one of our big projects right now Ian Linden: since we wanted it out for 1.7 Ian Linden: it's still coming along nively Ian Linden: nicely Ian Linden: HTML on a prim is into the "next thing" category, so a little farther out Ian Linden: new rendering system is still way out there, as we've slowed down on that Ian Linden: to fix problems w/ the current version Ian Linden: mono is anybody's guess Ian Linden: we have a working prototype Ian Linden: but it's going to take a while to hammer out the bugs and get the performance where we want it Ian Linden: all of these thing, ultimately, we come out when they're done Ian Linden: we've had poor luck giving out release dates in the past Ian Linden: so I won't embarass myself by repeating past mistakes Ian Linden: next? Jeska Linden: Mulch Ennui: regarding the " memory usage" bug you just mentioned, what are the symptoms of this bug in a stacked server situation? Ian Linden: I haven't studied it in 1.7 enough to give a good breakdown Ian Linden: but the main symptom will be a rapid drop in performance for the same number of avs Ian Linden: and the same general load Ian Linden: the first thing that happens is, the autosave process (which eats extra memory) Ian Linden: will start impacting the sim Ian Linden: so you'll see these hourly drops in framerate which don't last very long Ian Linden: the next thing that happens is, the autosaves from the OTHER sims start doing it Ian Linden: so then you have multiple drops/hour, on a regular schedule Ian Linden: the end stage is, all sims on that machine slow down because they're swapping Ian Linden: it's not pretty, but it doesn't often get that bad Ian Linden: I won't deny that there are memory leaks in the sim code Ian Linden: well, here's the thing Ian Linden: along with the legitimate cases of this occurence Ian Linden: there have been alot of problem blamed on "stacking" that really aren't a result of stacking Ian Linden: so it probably sounds a bit contradictory in the end Ian Linden: since we're respoding to different complains Ian Linden: and the day when there are no problems, I can retire Ian Linden: I doubt it Ian Linden: I don't think people know the best ways to cause a sim to bloat Ian Linden: I know I don't Ian Linden: yes it would but there are logistical problems with upgrading 900 machines Ian Linden: and it wouldn't realy solve the problem Ian Linden: I'd rather fix the leaks Ian Linden: honestly if a griefer could consistently cause a sim to bloat up to a huge size quickly Ian Linden: it would really help our debugging efforts Ian Linden: as it is, most sims takes a long time to grow, if they do at all Ian Linden: which is why the problems intensify which we haven't done a patch in several weeks Ian Linden: with 1.7 there seems little risk of that for a while though Ian Linden: next? Jeska Linden: Fizik Baskerville: the current island layouts are a total mess, is there any plans to create a more ordered and navigatable system ? Ian Linden: well we've certainly talked about ways to do that Ian Linden: and I think most of that got drowned in more immediate problems like fixing the sim list in the map Ian Linden: I don't think we arrived at anything really solid Ian Linden: so we'd like to hear your ideas Ian Linden: but yeah it's not an easy problem Ian Linden: since geographic location has alot to do with identity Ian Linden: next? Jeska Linden: paulie Femto: How is the program for providing offshore "void" sims, attached to estates, coming along? Ian Linden: daniel? Cyn Linden: I can take that Cyn Linden: We are ready to do some more beta testing with void sims Cyn Linden: You can email me for details Jeska Linden: Nexus Nash: what % of load does the SL software take on the servers? Ian Linden: all of it Ian Linden: all of the time Ian Linden: which is why SL uses 220kW :( Ian Linden: next? Jeska Linden: Surreal Farber: heh.. in regards to previous question on sim location... aren't sim adresses dynamic now, so moving doesn't change your LMs? Ian Linden: if we do it right ;) Ian Linden: but that's not a very smooth process Ian Linden: if it actually works at all, I'm not really up on that one Ian Linden: because landmarks are actually still referencing a location Ian Linden: when a sim moves we have to update them Ian Linden: which isn't very efficient Ian Linden: so yes/no Ian Linden: nexT? Jeska Linden: We've got time for one or two more questions from the queue. Jeska Linden: Nyteshade Vesperia: Are there plans to allow people to run sims using their own hardware? Ian Linden: not in the near future Ian Linden: long term there are various business scenarios in which that could happen Ian Linden: but there are a TON of issues with that Ian Linden: not least of which is the implicit DRM [digital rights management] embodied by the permissions system Ian Linden: which wouldn't work in that scenario Ian Linden: paulie: I dunno, our server building has 50,000 square feet of empty space Ian Linden: I can fit alot of sims in there Ian Linden: next? Jeska Linden: Anshe Chung: When you "sell" us so-called "land" or "sims", do we obtain something that maps to one certain value of computing resource that improve as technology improves and hardware price drop... Jeska Linden: ...or did we buy into something that continuously devalues through technology improvement. Will you pass on hardware improvement to land owners? Or will our "land" be devalued? Or will you use advances to just decrease your costs and improve profit Jeska Linden: margins.... I paid 1250$ for one sim. Does this mean that in 2 years I still own 1250$ worth of simulation power? Ian Linden: as stated before, we will upgrade old server, so in theory if you keep a sim for 10 years Ian Linden: you won't end up w/ a 10-year-old sim Ian Linden: I'm not eager to comment on what our pricing will be in the future Ian Linden: since we don't have any real plans Ian Linden: but i'm sure things will change Ian Linden: I like paulie's answer Ian Linden: I think the question is, are paying less for our servers while charging residents the same amount Ian Linden: our per-cpu purchase cost goes up and down Ian Linden: but stayes pretty stable Ian Linden: note that each class of sim is a bit faster than the last Ian Linden: 5 years from now, who knows? Ian Linden: but note that mainland sims start at $1000 Ian Linden: I would expect to continue to pay a premium for private islands Ian Linden: we could sell for $1 if actually putting the thing online were free Ian Linden: which it isn't right now Ian Linden: even if the hardware were free Ian Linden: I guess, long term, expect that we'll try to keep the prices fair Ian Linden: we're not interested in gouging our customers Ian Linden: it could happen Ian Linden: that latter is probably more likely Ian Linden: ultimately I personally think we're all better off if owning land in SL is a real investment Ian Linden: and if the price remains stable Ian Linden: next? Jeska Linden: Ok, last question -- Wayfinder Wishbringer: Jeska, I think a major question that is underlying here is why did LL decide to stack servers without informing/asking the clients who pay for those servers? Wouldn't an inexpensive dual-core 1 sim/1 server ha Jeska Linden: (thanks) Ian Linden: well the main reason we did that was to save space and electricity Ian Linden: well, again, we've been stacking for a long time Ian Linden: and I think generally the results are good Ian Linden: because no one thinks about the other case Ian Linden: when you have 4 sims one big machine, if 2 sims are small and light, there's more free room for 2 heavy ones before there are any problems Ian Linden: likewise they all can share their various caches Ian Linden: and are quicker to deploy updates to Ian Linden: so I think that stacking really is a net positive Ian Linden: and we're quite sure that the performance is good, as long as the sims aren't too bloated Ian Linden: and they have to get pretty darn bloated before things start slowing down Ian Linden: so I think what we all really want is for us to fix our memory leaks Ian Linden: I will say, BTW, that this wasn't an obvious outcome Ian Linden: when we first tested 4 sims on 2 dual-core machines it was a disaster Ian Linden: the performance was terrible and we never would have put that in production Ian Linden: but we were able to resolve the issue by moving to a 64-bit kernel Ian Linden: and enabling NUMA Ian Linden: which brought the performance back Ian Linden: so we did consider the results carefully Ian Linden: ok, I guess that's all the time we have but I can hang around here for a few more minutes.. Ian Linden: I think that's the "void" sim question from earlier Daniel Linden: Thanks Ian! Cyn Linden: Thanks for coming everyone Jeska Linden: Thanks for the good questions ;) Cyn Linden: Thank you Ian and Jes! Ian Linden: I think probably there's been some internal confusion about the nature of memory leaks Ian Linden: and again, we want to avoid creating a sim bloat boogey-man that gets blamed for everything Ian Linden: I will post some server specs on the sim stats wiki Ian Linden: but believe me, you're better off sharing a class 4 machine with 3 other sims Ian Linden: than alone on a class 2 machine Ian Linden: Mulch: I think the jury's still out on that one Ian Linden: cua: there's clearly a need for better script messaging and data storage Ian Linden: cua: lots of stuff in the works there, but I won't want to announce it yet because the specifics are still in flux Ian Linden: cua: well that's a new 1.7 bug I'm sure Ian Linden: cua: we'll look into that Ian Linden: Mulch: I'd love to hear what the new scheduling has done to your scripts? Ian Linden: Mulch: I they just slower, or is it something else? Ian Linden: wayfinder: no clue Ian Linden: what I'd really like is a copy of a script that broke w/ 1.7 Ian Linden: or an object Ian Linden: which I can give to QA Ian Linden: alright, I'll refer the bug hunters to racerx Ian Linden: a bug w/ a the havok interface is certinaly possible Ian Linden: ok Ian Linden: anything else? Ian Linden: I though this was going to be hard Cyn Linden: OK guys! Ian Linden: thomson, check Cyn Linden: I dunno about you but I have telehubs to set :) Cyn Linden: :) Cyn Linden: Thanks again everyone for coming Cyn Linden: Anytime Ian Linden: k, thanks folks, I'll have some sim class stuff in the wiki page I sent out earlier up in a little while Cyn Linden: Haven't decided yet Cyn Linden: I will keep you informed, and you can always IM me Ian Linden: wayfinder: mostly better higher Ian Linden: but we put some class 3s in the old class 1 slots Ian Linden: just to confuse everybody Ian Linden: well, IP addresses are a limited resource Ian Linden: but I agree we could have done it better Ian Linden: it didn't seem relevent at the time Ian Linden: and it won't be if we ever get the class info into the about box Ian Linden: Mike: maybe Ian Linden: sure Ian Linden: it could be in theory Ian Linden: there is only one Ian Linden: wayfinder: BUT Ian Linden: hah, still plain old pata Ian Linden: but the newer disks are quite a bit faster than the old ones Ian Linden: and, more importantely, the sims really aren't I/O bound Ian Linden: for one thing, the IO is threaded Ian Linden: so IO won't block the server or slow it down Ian Linden: it just means that, if it is being slow, assets etc will load more slowly Ian Linden: but we don't see that much, even on the class 4 machines Ian Linden: Wayfinder: basically it means that all the reads/writes happen out of step with the main process Ian Linden: wayfinder, so you're not waiting for it the way you do with most desktop apps Ian Linden: sortof Ian Linden: the real point is, class 5 will probably not have extra-fast disks Ian Linden: because there doesn't appear to be a need Ian Linden: whatever the next class is ;) Ian Linden: it's probbaly a year out Ian Linden: well, the asset servers are a different story altogether Ian Linden: THEY have the higher-end SATA disks, and LOTS of them Ian Linden: doubt it Ian Linden: more likely, if we want to provide access to that sort of data, it'll come in a web service Ian Linden: actually, the asset servers are using a very weird proprietary filesystem Ian Linden: that does striping in a way that isn't the same as raid Ian Linden: sortof Ian Linden: apache on top of the clustered filesystem Ian Linden: wayfinder: yes, a way to profile your script load and deputized control are important Ian Linden: but remember we have to balance that against, say, the lots and lots of people who still have frequent crashes Ian Linden: right Ian Linden: don't think we're not aware of the problem there Ian Linden: it's frustrating that we can't address this stuff more quickly Ian Linden: we've been planning for a long time Ian Linden: but, you know, people voted, and havok 2 was at the top Ian Linden: so that's what we're working on Ian Linden: yeah Ian Linden: it's supported alright Ian Linden: havok 3 is mostly just havok 2 w/ ragdolls Ian Linden: I didn't think so, but I won't argue w/ it Ian Linden: I'm not heavily into the havok stuff Ian Linden: all I know is that the decision was to skip it for now Ian Linden: ah Ian Linden: well then I just don' tknow Ian Linden: ah, region crossing Jeska Linden: Ok, thanks again for coming out everyone. Ian Linden: region crossing is always a release away from being fixed Ian Linden: someday we'll get it right Cyn Linden: :D Jeska Linden: Bye everyone! Ian Linden: k, thanks all, I'm off Cyn Linden: See ya guys!
|
![[RDF Data]](/fct/images/sw-rdf-blue.png)