03 February, 2012

many opportunities come rolling off your lap

I submit that your overall approach to fixing this problem of back-end instability and griefing is fundamentally flawed, for these 3 reasons:

1. There is no load detection capability available to developers. This means that systems cannot play nicely with the current state of throttle potential, and therefore any algorithmic attempt to stay beneath the throttle is nothing more than a hopeful guess.

2. There is no load detection capability available to users. A person has no way of deciding "Oh, I shouldn't use X or Y right now because they'll cause my throttle to be hit."

3. The penalty for hitting the throttle is ridiculously severe. The thought that a subscriber sending out messages too fast could cause a clicked landmark giver to stop working ... is mind boggling. Explain to me how giving inventory in response to a user's click can ever, in any universe, be considered griefing.

I hope and pray that I've misunderstood your throttling approach.
~DavidThomas Scorbal, maker of the very popular PoseAnywhere HUD, as well as other HUDs, scripted vendors, and scripted animation vendors.
Okay, so point by point.
  • 1. To be fair to the Lindens, they just announced this. They likely didn't have time to introduce throttling measures in the scripting language, nor give the tools for load detection to sim/store owners.
  • 2. While I agree the Lindens should have put such measures into place--or at least thrown some brown-bag meetings to developers and users while they were working this out--they felt they didn't have time to do so. This, again, may be something they're planning to introduce.
  • 3. "Ridiculously severe" is a very good descriptive for what's happening, though. Because of some undefined griefing potential, which they felt was slowing the entire game down, they rushed to get this out, without, I think, considering the merchant cataclysm awaiting them. But at this point, the Lindens aren't going to back down. I know some makers of vending and subscription systems are working like crazy to get fixes out; that's good. But in the meantime, it's causing more resentment than joy, and is yet another strike to the heart of the trust issue, where the Lindens are concerned.
I'm not entirely sure if the Search Engine Watch blog released Axi Kurmin from her reporting duties; I'm hoping she's just been too busy to breathe. But I'd love to know her take on this issue.

While I don't agree with everything Ciaran Laval says in his take on the issue, I think he makes one remarkably valid point--the issue is in sending individual inventory items to large groups of people, be that inventory bit a notecard, a texture, a landmark, or an object. What if each notice was simply that, an IM notice, that gave a typed URL for an announcement page?

For example: while Aki Shichiroji doesn't always do this, for the most part when I get a notice from her, it's just an IM, that points me towards her blog. And while--often--my IMs are capped by the time I get into world, thus "losing" her message, the IM still comes to my email. Which makes it quirkily easier to just click the link from Gmail and go directly to her blog to view new product releases.

Now, does every merchant on the grid agree with establishing external (or Second-Life branded affiliate internal) pages for product releases? Probably not. But each send to each customer wouldn't hit the wall of llGiveInventory throttling. And that would be a good thing, right?

Kelly Linden again:
David: There already is a 2 second sleep on llGiveInventory. The systems that are hitting the new limit are using many scripts to get around that sleep.
And David in reply:
Right so, how does it make sense to silently fail, instead of increasing the delay per user, if needed? I mean, you're going to have the same problems with the back end capacity if the user base grows and more people use llGiveInventory().

I would bet my life that if you polled SL scripters, you'd get 99.9% voting for an extended sleep period on throttle, over a silent failure. Surely you understand there is no way to detect or recover from this failure - and even following the "recommended" timing, is no guarantee that the throttle won't be hit.

Oh gosh, and just for laughs, how about stopping the abusers? Forcing this work onto us is just cheap, alienating and really discouraging to scripters.
Again, he's got a point--several, actually. First, he's right on the silent failure. When we send too many IM messages, we get a warning about writing too often. It's annoying, but it forces us to pace communication. When we send an item to someone in busy mode, we get notification back that the target did not receive our inventory send. That's also annoying, but again, it paces us. And entire product lines have come about due to this second "feature"--the system in place at Alchemy being one of the best, in my opinion. It sends a notecard to me, but if I'm not online, it 'sleeps' the return send request. Likely Ms. Cyannis gets a little missive that I'm off the grid, but more importantly, I get a polite little message that, because I wasn't online for the last send, it will try again later.

This, nearly always, is enough to get me to log in, even currently when I have only forty-two Lindens to my name, and have for nearly a month. Had I been possessed of more, I likely would have gone to the store and perhaps bought the item advertised, or another item--and in days past when I've been more flush, I've done exactly that. Her system works, and while it's not impossible for her system to hit that same llGiveinventory throttle, I'd imagine it does so far more rarely.

I'd also like to pull one line out from the above message and repeat it, because it sounds vaguely important:
Forcing this work onto us is just cheap, alienating and really discouraging to scripters.
I would slightly restate--I think it's really discouraging to scripters, builders, merchants, makers, and casual residents on the grid receiving no notice from their many subscription stores. While yes, for the last few days it has eased the announcement glut that's sent to my email, that's not a good thing--not if I want to peruse new products, go to new places, hear about new events. Because, again, hitting this limit now results in silent failure--and, in at least a few cases, the subscription kiosk happily churning away, sending out messages that are auto-muted from the moment it hit the inventory cap, and never once telling either the owner of the kiosk or the subscribers that this was happening, at all.

That's not good. That's kind of the definition of not good.

I wandered over to Crap Mariner's blog. I figured she'd have some perspective on it, and I wasn't wrong.

(Let me take a step back and explain the "she" thing. Gender, perception, identity, these things are fluid--and not just on the grid. But I tend to fix on the person as first encountered. I first encountered Crap as female. That has fixed my perception. I'm not saying my perception's the correct one--frankly, the only "correct" perception, in that tone of voice, is Crap's, flat out. But there, explanation, we move on now.)

There's not a large section of mention, but what there is, is worthy of pulling out here for added emphasis:
Not that I'd ever come close to the llGiveInventory() throttling surprise that the Lindens are throwing at people sending messages and notecards, mind you, but it seems that whenever someone comes up with a clever workaround to the unreliable core technologies of SL for the benefit of content creators and performers, the Linden developers work hard to close those loopholes faster than fixing the fundamental problems.
Crap and I are both cynical where the Lindens are concerned, no surprise there. But when the people who helped you build your platform, expand it, explain it to other people, and blog about it--and I don't think anyone can deny Mariner's one of those people--well, when those people start eyeing the exit doors? Not that the Lindens are going to ever realize this, but hey, it might be a heads-up from the universe--theirs or any other--that maybe they've got some wrong ideas. And just maybe they might want to fix them.

Not that we haven't said this before.

Jumping ahead, just a bit, I ran down the post mentioned in Mariner's entry, Notes from the Trough of Disillusionment. This is a concept developed by industry analysts Gartner, who detail several stages on the IT map:
  • the Technology Trigger (the breakthrough event, the product launch, the [usually press-based] discovery)
  • the Peak of Inflated Expectations (the press frenzy that follows, complete with over-enthusiastic embracing by early adopters, and the unrealistic expectations of total technology transformation that soon will not live up to the hype)
  • the Trough of Disillusionment (when the product or the event fail to meet expectations, and are then dismissed as 'dated' or 'unfashionable'; this frequently leads to press coverage of the product or the event as 'formerly great' or 'once-valid')
  • the Slope of Enlightenment (even though the press, for the most part, has run away, some businesses/individuals still use the product or event, and spend their time experimenting with the ways it does work instead of trumpeting its flaws)
  • the Plateau of Productivity (the product or event, through time, experimentation and longevity, is realized to be a good and helpful thing; this can also mean the original technology or software has been largely rewritten and recoded to be more stable, more reliable, and better adaptive)
So why does Iggy say that Second Life is in the Trough of Disillusionment? Look more closely at what that means in the cycle of hype:
  • Second Life has largely failed to meet user, press, and business expectations.
  • Second Life is now covered by the press, if it's covered at all, as a laughably outdated relic, or worse, a predatory environment filled with the most perverse things the reporter can imagine (cue: accusations, now constant, of pedophilia being rampant on the grid; it's not, but that doesn't stop the press from saying it).
  • Second Life, at this point, has gone beyond loss of users into actually hemorrhaging users and businesses. It's wounded. It is weakening. Combine this with the loss of owned sims and we get a virtual world that's not only getting smaller due to fewer users, but is actually getting smaller in terms of the amount of owned sims.
  • Inside Second Life, the bright spirit of exploration and discovery has dimmed for many. "It's not as fun as it used to be" is now an accurate statement, not an inflated one.
  • Also inside Second Life, we seem to have moved from Lindens and residents operating in concert, to Lindens and residents actively at war, with neither side (generally or effectively) understanding the other.
Okay, none of that's good. But what else? Well, there's his list of factors that contribute to Second Life's loss of content and customers. Some of the points I found particularly relevant:
  • the educational disaster. While there are pros and cons to education in SL, from beginning building classes to online medical training, the decision by the Lab to end discounted pricing for educational and non-profit customers was pretty lethal. A lot of universities left in October; some few waited until January of 2011 to leave, hoping the Lindens would change their minds. (Pro tip: the Lindens never change their minds.)
  • the rise of competition. It's not just OpenSim, there's now more than a few alternate virtual worlds rising. Whereas in 2009, there was zero competition for the Second Life user experience, now there is. There are former residents of SL who've moved lock, stock and pixel barrel to OpenSim, HyperGrid, or another virtual platform; some other users have left Second Life for places like World of Warcraft, even though the full socializing aspects are missing, because if they're going to be screaming at the screen over a game, they'd rather do it trying to fight evil, not the Lindens.
  • the fact that logging into SL these days requires major upgrades that, still, most people just don't have. (While I understand that, to be competitive, SL wants to equalize the 'look' and feel of the game with semi-equivalent MMO game worlds, what that has meant to the end user is a game that perpetually changes the rules for access. At this point, to use everything SL currently offers, with the latest version of their software, the end user pretty much needs to already own a dual-core computer system. If not, at some point, the experience will start to limp for them. (Pro tip: most users of SL don't have more than a single-core computer board. This has led to widespread dissatisfaction with the additional graphic demands on users.)
  • this one I'm lifting from Iggy directly: The stability of the virtual world and the need for frequent client upgrades (both largely resolved today). Mainly because my perspective is different, and I'd state it as the instability of the platform, and the need to hop about sometimes once a week searching for new builds of third-party viewers, to ease the insane memory leaks and hopefully, find a stable browser that won't crash every 60 seconds.
  • another one I'm lifting directly from Iggy: The relatively slow pace of empirical evidence for teaching effectiveness in immersive 3D environments. Ironically, this has emerged in 2010 and 2011, but by them many campuses had decided to focus more on mobile technologies and social networking for teaching. I'd only half-disagree--I think there are good and valid reasons to teach in SL, from the simple (learning SL building, for example, is kind of impossible if you can't build along with the instructor) to the arcane (sociologically, people seem to learn better in virtual spaces if they can see the instructor's face, or their fellow students' faces. We're strange that way).
  • and, ultimately, the combination of the loss of the educational discount, with the lack of strong evidence that SL learning was inherently "better", with the long-held opinion (mostly by the press) that people just don't learn in "games", meant very, very few non-profits or educational concerns wanted to be affiliated with Second Life, or Linden Lab.
And nothing on that list is good. It's pretty much a case-by-case point for why Second Life is failing so utterly right now. Can they turn things around? Well, obviously, but the Lindens' chief problem is they keep pulling boneheaded moves like Ursula/Zindra, the OpenSpace debacle, the RedZone debacle, and the current inventory throttling catastrophe...among others...and never seem to realize that these incidents are retained in memory, and it becomes that "one more thing". Sooner or later, if the Lindens don't snap out of it, their world will fail, because they'll be the only ones on the grid.

And in the meantime, the world becomes emptier every day. How depressing is that?

4 comments:

Aki Shichiroji said...

Thanks Emily,
Yeah, I began using IM-only notices after becoming a bit frustrated with the lack of feedback and info about interested and active subscribers when I just sent out note-cards. Even though I'm nowhere near the point where I'd hit that throttle, I find it particularly useful to just include a link because many people tend to check their email while not able to access SL. I can track (in a broad sense) which promotion methods are working best for the expense, what items people seem most interested in, general locale, etc.

Since I follow up almost every product I post with an equivocal listing on the Marketplace, this often means interested individuals will at least check out those listings and keep them in mind for later if they don't pick them up outright. Folks can also explore other releases and news items, so as a whole I find utilizing external resources (such as my own blog, Plurk, Twitter and Marketplace) to be of particular usefulness.

I understand my usage isn't necessarily representative of the majority and I get that many merchants would like to send out group gifts or attempt to get around the IM throttle by sending out note-cards instead...but I do feel there are options, particularly since 1) gift delivery via group or subscription device tends to fail in volume anyway 2) offering a gift in-store only will encourage subscribers to not only pick up the gift, but shop as well. Many of the complaints on the JIRA could be covered by either sending out news via link in IM or setting out group gifts in-store, accessible only to a specific access list or responsive to an activated group.

I do find it unfortunate for those few who have genuine use cases for which there are few alternatives; I know Seven Shikami spoke up in the JIRA comments at some point and it's not altogether clear to me whether her business is being severely affected at this point, however I could see how it easily might be, given the broad adoption of her product. Various other content makers, such as breedable companies and such, may also be affected.

For those people, I do worry and wonder whether LL has thought things through and I do hope a compromise can be found.

Emilly Orr said...

This came in under "Unknown", but you're welcome. And yes, I do think much of this was because of the original subscription work-around in the first place. But the only reason Subscribe-o-Matic and other companies developed those technologies was the restriction to such a limited selection of groups. Fine if you're just a casual user of SL; not at ALL okay if you build, run a business, run a club, an art gallery, radio station, or simply shop in many places. Toss in job groups and land groups and that original limit maxed out in no time at all.

Even now, I keep one group shy of 42, only because it's easier for me--and that's after leaving many store groups, land groups and job groups behind.

Seven Shikami remains the chiefest injury of the new policy, and right now, there's NOTHING Shikami can do about it. With the current coding, which underlies all 7Seas products, once that throttle is hit, everything shuts down. Now, granted, that's per region, but still, say there's a fishing competition in the Rag Dollz store, the Avilion Forest fishing dock, and Charlestown, all on the same night. Assuming they're well advertised and get thirty participants each, each place will have outgoing requests to the main server on the land every, what, twenty-five seconds or so? Let's say two every minute. So that's about 144 requests every hour, on average, per person. (Assuming they all pay attention and keep tossing out their lines.) Toss out the last twenty-four as being null results (requests that mean the fish "slipped away"), and that means 120 requests for won fish per hour, per person.

That means 3,600 requests are winging back to the players of that competition, which means, again, on broad average, 2,000 requests are going silent and failing. No fish.

Okay, fine, that's one competition, one region. But that's happening over the entire region--all privately held servers, all servers at other shops on the same sim. Even the places that aren't running contests at, say, Rag Dollz, are going to shut down.

Because Seven Shikami owns all the servers for fish. Why? Because someone hacked Seven's equipment a while ago, and individually-owned servers were vulnerable.

So yeah, for 7Seas it's a huge deal.

Aki Shichiroji said...

Sorry, I thought Google would grab my account name while I set up my account to work with Blogger, but apparently the system doesn't automatically fix names that weren't defined at the time of comment postings. Hopefully this reply will work better/be more clear.

Emilly Orr said...

Generally it should do so, especially since Google also owns Blogger now, but weirdly, it doesn't always. Not sure why. And the reply was fairly clear before, but it does add more understanding to know you're the one who wrote it.

(Also, while we're here: glad to hear that Kitheres is back as a brand. Looking forward to what you turn out/revive.)

I really think that's the best option, and it wouldn't be that hard to script an IM-only system. A lot of us have our IMs mailed out because in-world conversation in general has been flaky since...always? So it's habit for many of us.

But again, it's dealing with nostalgia, investment and habit--people are accustomed to using subscription services, and they don't want to look into other options, because it worked before, why doesn't it work now? Especially large groups (I'm thinking specifically places with over 10,000 members, active or not), wherein the investment comes in--not only did they buy the system, but at those member levels, they generally pay a monthly fee as well.

But with the throttling, subscription services are going to get a lot slower, or go the way of the dodo. And I still haven't found mention one was of what security issue the Lindens were fixing. (Though frankly, that question may never be answered.)

I've got a three day headache and it's all in my head

It's the 30th of March. One day before Ostara. And there's been a lot of...well. Conversations like the one below. [18:43] Emil...