In some current code, I have been using
base64-encode(generate-uuid)
to get a character GUID to use as a record identifier. In Windows 7, it works fine. In Windows 8.1 it is coming up with duplicates. In particular, this test program:
define temp-table ttKeys no-undo
field chID as character
field inCnt as integer
index ixID as primary unique chID
index ixCnt as unique inCnt.
define variable pinCnt as integer no-undo.
define variable pinLimit as integer no-undo initial 400000.
repeat while pinCnt < pinLimit:
create ttKeys.
assign
pinCnt = pinCnt + 1
ttKeys.chID = base64-encode(generate-uuid)
ttKeys.inCnt = pinCnt
.
catch eobSysError as Progress.Lang.SysError :
define variable inWhich as integer no-undo.
message substitute("Unexpected Exception: &1", eobSysError:getMessage(1)) view-as alert-box.
message substitute("Count = %1", pinCnt) view-as alert-box.
end.
end.
if pinCnt >= pinLimit then message "Success" view-as alert-box.
will pretty consistently produce a duplicate key, although sometimes it is 330000 records in before that happens. With Windows 7, we have gone to a million with no dups.
Do I have some misunderstanding in thinking this should work or is the 8.1 behavior a bug. TS is working on it.
In some current code, I have been using
base64-encode(generate-uuid)
to get a character GUID to use as a record identifier. In Windows 7, it works fine. In Windows 8.1 it is coming up with duplicates. In particular, this test program:
define temp-table ttKeys no-undo
field chID as character
field inCnt as integer
index ixID as primary unique chID
index ixCnt as unique inCnt.
define variable pinCnt as integer no-undo.
define variable pinLimit as integer no-undo initial 400000.
repeat while pinCnt < pinLimit:
create ttKeys.
assign
pinCnt = pinCnt + 1
ttKeys.chID = base64-encode(generate-uuid)
ttKeys.inCnt = pinCnt
.
catch eobSysError as Progress.Lang.SysError :
define variable inWhich as integer no-undo.
message substitute("Unexpected Exception: &1", eobSysError:getMessage(1)) view-as alert-box.
message substitute("Count = %1", pinCnt) view-as alert-box.
end.
end.
if pinCnt >= pinLimit then message "Success" view-as alert-box.
will pretty consistently produce a duplicate key, although sometimes it is 330000 records in before that happens. With Windows 7, we have gone to a million with no dups.
Do I have some misunderstanding in thinking this should work or is the 8.1 behavior a bug. TS is working on it.
Base-64 encoding is case-sensitive and having two values that vary only by case is perfectly acceptable. So using caps() will induce collisions with other base-64 encoded UUID values and not allow you to recover the initial raw value. So this is not a valid workaround.
for example: the value 2000 encodes to "MjAwMAo=" : if you uppercase and decode "MJAWMAO=" you get an invalid value ( 0?0 ).
Jeff was suggesting using hex-encode instead of base64-encode. AFAIK, hex-encode is not case-sensitive, unlike base64-encode.
You could define the chID field in your temp-table as CASE-SENSITIVE and still use BASE64-ENCODE. Then it will depends on how the code is using it later (comparisons should then be made against the temp-table field for the comparison to be case-sensitive too).
Thomas did you see if the generate-uuid generated duplicates as well? It is most likely that the way the "random" GUID is generated that is at fault even before encoding is done.
I had to write something in the mainframe world as we did not have standard libraries to create GUID's and some of the suggested "random" methods did not work at all.
If I do MESSAGE GUID(generate-uuid). I generate a lot of numbers which is in format:
XXXXXXXX—XXXX—XXXX—XXXX—XXXXXXXXXXXX
but the first 3 groupings are exactly the same, which is not EXPECTED!
Is it possible to get the code that generate these UUID and GUID?
the fact that you do not expect the first 3 groupings to be the same does not mean it is wrong.
uuids are made up of various parts. one of those is a proxy for location, often derived from an ethernet mac address. it is the whole things that is supposed to be unique, not the parts that go into one.
Gus, I do understand that. :-) my gut feeling is just that the smaller the "random part" the greater the chance of duplication.
Also the V1 GUID Algorithms tend to use MAC address and Time as part of the value but these are indicated by a 1 as the first digit/character on the 3rd grouping. These ID's are all Version 4 which are PSEUDO random number (number 4 in first digit of the 3rd grouping.
And the numbers I generated today all had exactly the same values in the first 4 groupings. The UUID only changes from position 23 onwards.
Not saying it is wrong, but merely asking id the GUID's could not be a bit more random?
But I will go write some code to populate a table with extremely large number of GUID as primary keys and see how often duplication comes up.
I am just interested in knowing how the IDs are created and if it could be improved upon. :-)
@alex: if you are getting duplicate GUID's then that is a bug and has to be fixed.
if you are not getting duplicates then it is working as it is supposed to and does not require any improvement. uniqueness is all you have a right to expect.
Switching to a case-sensitive field has fixed things for me. Youssif, who is working with me on my TS case, said:
I just tested the following two version (one tests values generated by GENERATE-UUID calls, the other tests values generated by the BASE64-ENCODE(GENERATE-UUID) calls. My count was set to 10 million records on Windows 2008 R2 under 11.3.2 without duplicates:
We still have no idea why, without the case-sensitive, we were getting dups on 8.1 and not on 7. But, with the case-sensitive I get no dups up to 1 million in multiple passes.
Is Progress calling an OS function to generate UUIDs?
To wrap this up, thanks to the thoroughness of Youssif Shanshiry ...
One can index a temp-table by a raw, but not a database table, so one *must* convert to character to use a guid as a primary key. There are three methods:
base64-encode(generate-uuid)
Yields a string 24 long, the last 2 of which can be discarded. The string is case-sensitive.
guid(generate-uuid)
Yields a string 36 characters long which is not case sensitive.
hex-encode(generate-uuid)
Yields a string 32 characters long which is not case sensitive.
So, you makes your pick.
Thanks for the wrap up Thomas, that is quite a relief.
We are heavily relying on GUID(GENERATE-UUID) and use the generated values as globally unique keys for merging records in distributed databases.
In the beginning of this thread I was shocked to find that this might not be safe in hindsight when the indexes are not defined as case-sensitive (which the are not). But now it seems to be safe after all, as the documentation has always suggested.
So the only problem appears to be with base64-encode(generate-uuid).
No need to use GUID(GENERATE-UUID), just GUID is enough. Besides a separate guid-generator wrapper class / method or whatever you call it can be handy. The 36 char guid as primary key has a performance trade-off that could be improved in the future. See f.e. www.codeproject.com/.../GUIDs-as-fast-primary-keys-under-multiple-database and the recent disussions on the clustered index / comb guid on the peg.
@peter: there is no problem with base64-encode(generate-uuid) either. the problem is with how the result of base64-encode is used.
Meaning the case-sensitive part or something else?
meaning that the base-64-encode function returns a mixed-case value (as it should) and that value is then misused by treating it as case-insensitive, which leads to errors.
OK, fine. I personally have no problem [now] recognizing that the cost of the shorter string is needing to be careful about being case-sensitive everywhere it is used.
Interesting article. Does it also apply to OpenEdge when it says "most databases store rows in what's known as a clustered index, meaning that the rows in a table are actually stored on disk in a sorted order, usually based on a primary key value"? As far as I know this is true when loading an OpenEdge database but does OpenEdge store records physically on disc based on the primary key?
See recent discussions on the peg.
-------------------------------------------------------
On 6/4/14 2:39 AM, "Stefan Houtzager" wrote:
>Has the possibity to mark an index as clustered in a progress db been considered bij psc by the way?
it has been discussed in the past but not seriously considered as there have been higher priority things to do. afaik, no one has mentioned it in recent times.
regards, gus
--------------------------------------------------------
Kind regards,
Stefan Houtzager
Houtzager ICT consultancy & development
www.linkedin.com/in/stefanhoutzager
Sorry, don't have those posts and the most recent on the website is from 2009 ...
Sent off line.
I can recommend the peg. You are not obliged to pay anymore and there are very interesting discussions there. Moreover the peg is independent (and that has proved to be important).
I can recommend the peg. You are not obliged to pay anymore and there are very interesting discussions there. Moreover the peg is independent (and that has proved to be important).
Flag this post as spam/abuse.
It's a very good place. You remember the incident where a psc employer sent me an invitation off the line to call him at his office, questioning the relevance of the content of some of my messages on the architecture forum here. A clear attempt to censor me. It never happened again (after I made the reaction public on the peg, but I don't know if that was the cause).
I do remember. And I am with the many folks that does very, very well understand Jean's action there.
If "agent_008" with no further personal information would post uncommented (so absolutely no added value) links to competitors websites, I would probably do the same.
Still posting them. Postgres f.e. is a competitor too. You can learn from competitors. I have the impression you are almost a progress employee, of the wrong kind, more "loyal" than some psc employees themselves (cool aid drinker others would call you). You get me a bit furious again Mike, you're so biased and political. There a couple of other messages of people that were *not* with you at that time. No added value for you does not mean no added value for everyone. You're big, but not that big. And I'm not afraid of you like others maybe.
I'll refrain myself of further reacting on all negative reactions that will follow here. The matter has been handled already.
-----------------------------------------------------------------------------
Tom Bascom via peg.com Jan 15
to peg
It was not a "link only" post. It was clearly labelled and even a cursory look at the link itself clearly shows that it is probably relevant to the topic.
The post (community.progress.com/.../8809.aspx) in its entirety:
> Making the best of MDD
>
> read this interview with the mendix cto:
>
> modeling-languages.com/coffee-johan-den-haan-mendix
I often follow Stefan's links and have found some of them quite interesting. I appreciate that he doesn't spend a lot of time trying to influence my opinions about the content of the link but rather just says what it is and puts it out there as something to consider.
I didn't find this one particularly fascinating but I have a hard time seeing how anyone at Progress could see it as inappropriate.
-------------------------------------------------------------------
Gus Bjorklund via peg.com Jan 15
to peg
On 1/15/14 3:35 PM, "Tom Bascom" wrote:
>I have a hard time
>seeing how anyone at Progress could see it as inappropriate.
me too.
regards,
gus
---------------------------------------------------------------------
Tom Bascom Jan 15
I get a lot of spam. That post doesn't look anything like spam. I'd share some but PEG, as we all know, aggressively filters spam. Notice how even PEG has not filtered this message.
Let me refresh your memory. The post (community.progress.com/.../8809.aspx) in its entirety:
> Making the best of MDD
>
> read this interview with the mendix cto:
>
> modeling-languages.com/coffee-johan-den-haan-mendix
The post title says that it is about "MDD". That topic seems, to me anyway, a perfectly appropriate topic for the forum.
The URL clearly and in plain English says that it is about modeling languages and "mendix". Which matches Stefan's description of what he is asking people to read.
The content comes from outside progress.com. Maybe that bothers someone at Progress but, personally, I value opinions and ideas that differ from my own. Even when they are wrong. If nothing else they give me an opportunity to enumerate the reasons why they are wrong ;) But I like it better when I see a new idea or learn that, actually, I was wrong and some other approach has advantages.
-----------------------------------------------------------------------------
Tom Bascom via peg.com Jan 15
to peg
Censorship isn't just removing content. It is also about discouraging people from saying certain things. Having a phone call to discuss "alternative approaches" is kind of intimidating if you ask me. It sounds like corporate-speak for "clean up your act or else".
On the surface it all sounds very innocent and harmless. Until your boss comes around to your desk. (Maybe you've never had someone "reach out" to your boss because they didn't like something you posted online. Clean living will do that for you...)
Now that isn't exactly what is happening here -- but it is easy (for me anyway) to see some parallels.
Not all censorship is bad. For instance I'm perfectly ok with Greg making sure that we don't get spam. And I think it is perfectly ok for Progress to remove any crude and unprofessional sorts of postings. I'm even ok with them taking down thinly disguised competitor advertisements and such.
But I don't see this as even bordering on any of that. It merely mentions another company with products in what seems like more of a complimentary than a competitive niche and the article itself mostly talks about the advantages of that approach. Material which I would *hope* that Progress marketing and development people might want to spend some quality time thinking about. It's not my particular cup of tea but I can see how Progress might want to be thinking about that kind of stuff.
It's not much different than pointing out that certain companies with sailboats have some very interesting database features that might be fun to have in OpenEdge. If I posted something about that I wouldn't expect an invitation to discuss alternative approaches to my posting habits -- I'd expect something more like "great idea! we'll get right on it" or (more likely) "we thought about that and here's our approach..." or (most likely) "tell us about your use-case...".
Is it Friday yet?
-------------------------------------------------------------------------
no it's not Friday. It is time for you guys to get back to work.
The key issue, Stefan, is that you never contacted Jean to find out what he was actually thinking. You based your response entirely on what you thought he might be thinking, what motivations he might have. All of the PEG reactions were based on an idea which actually had no basis in reality. There was never any intention to censor., But, that is the kind of thing that happens when you assume what someone else means and don't do them the courtesy of at least listening to what they have to say.
PEG has has its incidents too, by the way.
The key issue is the mail I got from the man. If you can't see the intention you need stronger glasses. Just for the readers info so they can see for themselves:
"Hello Agent_008_nl,
I just saw your post in our forum about the Mendix CTO interview. Can
I ask you to contact me back so we can talk together on alternative
approaches as I'm far from being sure this post does add any sort of
value to our community.
My email address is <xxxx>. Thanks to email me back so we can agree on
a day/time fopr a call.
Cheers."
Read Tom's mails that I pasted. I'm not going to discuss with someone as biased as you, your priorities are clear.
It’s always fun to try to start a discussion with you. You just refuse serious arguments.
At the risk of being called a kool-aid drinker, I have found that Jean is one of the most open minded people I know.
I also sometimes rage against the machine, as does Mike (believe it or not!), but cannot complain about anything Jean has said or done.
Julian
[collapse]Reply by agent_008_nlRead Tom's mails that I pasted. I'm not going to discuss with someone as biased as you, your priorities are clear.
Stop receiving emails on this subject.Flag this post as spam/abuse.
+1, no kool-aid here... lots of ratm though, just can't see Jean as
being part of 'the machine' ;)
[collapse]On 06/18/2014 09:45 AM, jmls wrote:
>[collapse] From: jmls
> Post: RE: GUID or not?
> Posted in: OpenEdge Development
> Link: http://community.progress.com/technicalusers/f/19/p/10581/40544.aspx#40544
>
> At the risk of being called a kool-aid drinker, I have found that Jean is one of the most open minded people I know.
> I also sometimes rage against the machine, as does Mike (believe it or not!), but cannot complain about anything Jean has said or done.
> Julian
> On 18 Jun 2014 07:39, "agent_008_nl" wrote:
>
>
>
> RE: GUID or not?
>
> Reply by agent_008_nl
> Read Tom's mails that I pasted. I'm not going to discuss with someone as biased as you, your priorities are clear.
>
> Stop receiving emails on this subject.
>
>
>
> Flag this post as spam/abuse.
>
>
> --
> You were sent this email because you opted to receive email notifications when someone created a new thread.
>
> To unsubscribe[collapse] from:
> - ...only this thread, disable notifications at http://community.progress.com/technicalusers/f/19/p/10581/40544.aspx#40544.
> - ...all email notifications from Progress Community, navigate to "Settings", click on the "Email" tab, then under the "Email Configuration" section, set Send Notifications to "No".
>
>
--
m.edu
keep it simple
http://www.ganimede.ro
http://ro.linkedin.com/in/marianedu
medu@ganimede.ro
mobile: +40 740 036 212
skype: marian.edu[/collapse][/collapse][/collapse]
Already said: read Tom's serious arguments.
I'm not going to repeat endlessly and love to send links to others' arguments or copy them. I'm with Tom's and have nothing to add. I did not paste them to be ignored again by you and your team. And I hate soccer (not the game but the noisy beerdrinkers / sausagegobblers that piss openly on the streets after the game and claim "we won".
How hard are you trying to get everybody wrong? I never said you should enjoy Dutch soccer fans ;-)
I'm not asking for "your" arguments once more. But who is "my team" here?
Sorry, but I am bad at splitting the sight of the arena from the soccerfield. And the arena from the street. If I see a soccerplayer I hear the beerdrinkers scream and swing with their sausages and feel a bit lonely. Moreover I see those cheap frenchfrieshops with those boring pics of a soccerteam and the trophycabinet. That's not all. ;-) I'm not going to the psychiatrist for treatment. :-) A full barn with vitriol is not bad perse.
Your team here consists of those that support your opinions unisono on this occasion. I'm happy truth, if it exists, is not democratically chosen. Judges should not let their ears hang over to the street.
The point, Stefan, is that everything that anyone said, including Tom, was based on your interpretation of the email. During much of the discussion we didn't even have the text of that email. We all have said things, especially in email, that got read by the receiver with an intent different from what was in our mind at the time of writing. By not asking Jean what he meant, you are not giving him the chance to correct that misimpression, but rather are condemning him for all time for something he may well not have meant (and I know personally that it wasn't what he meant). Jean is very, very committed to improving the quality of these forums and works at it very hard. I think the results show in the increased registrants, visitors, and traffic. Never, ever, have I seen a case of censorship. Do you think you are that special?
[quote user="agent_008_nl"]
I can recommend the peg. You are not obliged to pay anymore and there are very interesting discussions there. Moreover the peg is independent (and that has proved to be important).
[/quote]
Let's stay with facts here: http://www.peg.com/about/
"Everyone who uses PEG whether via e-mail or via the net is expected to become a member if they can afford it. If you live or work in the USA, Canada, Western Europe, Australia or New Zealand and you are not unemployed, we assume you can afford it."
AFAIK Holland is part of Western Europe. So PEG is expecting you to be a paying member.
Are you going to say that Greg is now also part of my team?
Thomas & Mike, you make me weary. Nothing to more interesting? Now watch soccer, I hope Holland lost! And of course Germany even more. Kuppers Kolsch: lousy beer Mike!
What is more wearying than accusing a perfectly nice person, one who is working very hard to build this community, and claiming that he is doing bad things ... and making those accusations without even contacting him to find out what he means.
move on guys. go find something useful to discuss.
Like GUIDs? (Or not)
Thanks to Thomas I read the clustered index discussion on PEG and it answered my question.