FIM Custom Expressions inside Custom Expressions?

Recently, I needed to take Longitude and Latitude data that was given to me in the following format and break it into its individual components and then flow it out to AD.
Let's suppose the data looks like this:

"Point -10.1223 45.945"

I could just use the Left and Right functions to get out the Longitude and Latitude.

The problem was it could also look like this depending on the level of precision:

"Point -10.1223453333 45.945111113" 

So using the Left and Right functions were right out.

But I could use the Word function

Word(GeoData,2," ") gives me -10.1223 for the first row and -10.1223453333 for the 2nd row

Word(GeoData,3," ") gives me 45.945 for the first row and for the 2nd: 45.945111113

Suppose the data was slightly different a comma instead of space in between.

"Point -10.1223,45.945"

"Point -10.1223453333,45.945111113"

Now I can add another delimiter in addition to the space character I can put in a comma:

Word(GeoData,2," ,") gives me -10.1223 for the first row and -10.1223453333 for the second row

Word(GeoData,3," ,") gives me 45.945 for the first row and for the second: 45.945111113

Let's add another twist (which gets to how the data was really presented) and put the data inside parenthesis like so:

"Point (-10.1223 45.945)"

"Point (-10.1223453333 45.945111113)" 

But when I tried to do 

Word(GeoData,3," (")

I got an error "the function Word is not correctly formatted"

Then I had a brain storm! What about putting a CustomExpression inside a CustomExpression?

Bummer: "The function named CustomExpression could not be located" so no nesting of Custom Expressions!

So for giggles I decided to not start it with a customExpression

That worked!

So the lesson is that sometimes trying to use some of the special characters, like a parenthesis inside of a literal string confuses the CustomExpression parser but can work when put inside the parameter of the function call (not inside the Custom Expression)

I did indeed confirm that the parenthesis was the problem as it worked with other characters just not a literal parenthesis when doing a whole custom expression.

How many attributes can you have in the Metaverse?

Back in 2013 I published 5 posts about the Secrets of the Metaverse:

Parts 1-5:

1. What is the Metaverse?
2. How is the Metaverse data stored?
3. Is there a limit to how many Metaverse attributes I can have?
4. Has access to the metaverse gotten faster with recent releases?
5. How do I safely query the metaverse?
6. Added (Aug 5 2015): How Many Metaverse Attributes can I have?

The third post was about how many attributes you can have in the Metaverse in which I said that the mms_metaverse_lineageguid table limits us to 502 single valued non-reference attributes in the Metaverse. This is still correct but a client told me of a scenario they encountered where the lineageguid table prevented them from getting to over 450 attributes and they encouraged me to blog about how they solved it.

The issue can occur when you delete attributes from the Metaverse and then try to add more. If you exceed 502 single valued non-reference attributes that have ever existed in your Metaverse you will encounter this error unless you take some very specific actions at the database level. WARNING: these actions should be done under the direction of Microsoft Support so that your installation can remain in a supported state.

The client had deleted a number of unused attributes prior to adding the many attributes they needed and then hit a brick wall getting the following error in their application event log:

 0x8023042e (the table cannot be created because the row size limit was exceeded.):SQL: ALTER TABLE [dbo].[mms_metaverse_lineageguid] ADD [theirAttribute] [uniqueidentifier] NULL 0x80230629 (the specified metaverse schema has too many attributes).

First: Why does this happen?
As you can see from the error message when you add or delete a single valued non-reference attribute to the Metaverse the Synchronization Service runs an ALTER TABLE statement to add or delete a column and as famed SQL MVP and author Kalen Delaney states running "ALTER TABLE will not reclaim space." Her article is about altering the length of column but "Database Whisperer" Michael J Swart provides an example of removing columns and shows that ALTER TABLE just makes a meta data level change. So even though the column is not used anymore it is still taking up space in the table until the Clustered Index is rebuilt.

You can see how close you are getting by using the following query (I used Michael's and Kalen's queries as starting points):

USE FIMSynchronization
GO
SELECT  c.name AS [Column Name], column_id, leaf_offset, pc.is_dropped
 FROM sys.system_internals_partition_columns pc
    JOIN sys.partitions p
        ON p.partition_id = pc.partition_id
   LEFT JOIN sys.columns c
         ON column_id = partition_column_id
            AND c.object_id = p.object_id
WHERE p.object_id=object_id('mms_metaverse_lineageguid')
ORDER BY pc.leaf_offset
GO

Column Name Column_ID Leaf_Offset Is_Dropped

------------------ -------------   ---------        --------------

title                  99              1564           0
type                  100              1580           0
uid                  101              1596           0
NULL          NULL      1612           1
NULL          NULL      1628           1
testAttribute3  105              1644           0

In this example I have added three attributes and then deleted two of them (the first two I added). As you can see this leaves behind some open space in the mms_metaverse_LineageGuidtable and means that I would hit the limit sooner than I would expect.

If the biggest Leaf_Offset is 8044 then you are out of space to add more single value non-reference attributes (8044 byte offset+16 bytes =8060 bytes limit for a SQL row).

Normally, rebuilding the clustered index will reclaim the space for you. So you think no problem since you have followed my advice in FIM Best Practices Volume 1 and you use Ola Hallengren's scripts to automate index maintenance. However, the script will only rebuild if the index is more than 30% fragmented otherwise it will just reorganize it (which doesn't reclaim the space). So you could rebuild the clustered index by hand. Oops! The mms_metaverse_LineageGuid table doesn't have a clustered index -- so you have to add one. But then to return the database schema to its supported state you need to drop the clustered index. You can the clustered index on the object_ID column as this will be unique and not null. Then drop it.

 ONCE AGAIN: ONLY DO THIS UNDER THE DIRECTION OF MICROSOFT SUPPORT if you want to stay supported (and with all Syncs halted).

CREATE CLUSTERED INDEX [CX_mms_metaverse_lineageguid_object_id] ON dbo.mms_metaverse_lineageguid ( object_id)

GO

DROP INDEX [CX_mms_metaverse_lineageguid_object_id] ON dbo.mms_metaverse_lineageguid

The creating or dropping of a clustered index forces the rebuilding of any non-clustered indexes. Unfortunately that means that this happens twice. But that can not be avoided. For a big table that can take a while (tens of minutes for metaverse with hundreds of thousands of rows). Of course the pure DBA would prefer to create an appropriate Cluster Index and leave it, rather than drop it, but that would put me out of support by the MIM product group.

When I rerun the query I get:

Column Name Column_ID Leaf_Offset Is_Dropped

------------------ -------------   ---------        --------------

title                  99              1564           0
type                  100              1580           0
uid                  101              1596           0
testAttribute3  105              1612           0

You can see that testAttribute3 is now offset at 1612 and the other columns are gone. Now that space is available to use for the lineageguids of attributes. 

So that is how you reclaim the space so you can add more attributes.

MIM 2016 is now available

MIM 2016 is now available

MIM -- Microsoft Identity Manager 2016 builds on and replaces Microsoft's Forefront Identity Manager 2010 R2.

On Microsoft's site they include an introductory (2 min) video about Hybrid Identity but don't mistake that for the MIM UI.

So has anything been removed?

No. While the list of deprecated features are still deprecated none of them have been removed from this new version.

So what's new?

The first thing to call your attention is the focus on Hybrid (Cloud + On Premise) Identity. MIM can still manage on premise but is now even better equipped to work with Microsoft's Identity Management pieces in the cloud.

PAM -- Privileged Account Management. You establish a secure Forest (a bastion forest) with MIM deployed there and you request temporary membership in a role that then puts you in a group in the bastion forest that grants you elevated privileges in your main forest(s). Microsoft's take on this is different than others such as CyberArk which vault your privileged passwords and change them frequently.

SSPR -- Self Service Password Reset can now use Azure MFA (Phone Factor)

Self Service Unlock -- Use the SSPR mechanisms to authenticate and then unlock your account without resetting the password. Useful when you change you password but forget to update the cached password on your phone.

Certificate Management -- Win 8.1 client, no need to join to the domain (if using ADFS), Virtual Smart card, claims, events for troubleshooting.

Still an MVP but now DS MVP

I have been awarded the Microsoft Most Valuable Professional for a 9th time. I started off as an MIIS MVP (even though ILM had been released 4 months previous). Then I became an ILM MVP in 2008, then in 2010 it was FIM MVP (or was that 2011). Now with FIM changing to MIM and in an effort to reduce the administrative paperwork the Microsoft MVP team has every time MMS/MIIS/ILM/FIM/MIM changes names all FIM MVPs have become DS (Directory Services) MVPs. ;) Actually, they decided that there was enough overlap and dependency that it made sense to combine them. So now I am a Directory Services MVP

Big Data needs Identity in order to Act

At the 2015 Identity Summit Scott McNeely declared "Big data without Identity is not actionable"

Let's discuss.

Pulling from Information Week and IBM the Top 6 use cases of Big Data are:
1. Big Data Exploration
2. 360 degree view of customer
3. Information Security and Intelligence
4. Operation Analysis of data from Internet of Things
5. Data warehouse Augmentation/Optimization
6. Big Data Efficiency play (break down silos)

Big Data use case	How Identity makes Big Data actionable
360 degree view of customer	How can you understand the customer from 360 degrees if you don't know who the customer is? The trick here is linking their social media info to their account info -- talk about resolving ambiguous joins!
Information Security and Intelligence	If you plan to leverage social media to improve your fraud detection once again you must link that data to your customer's account. Without that you have nothing to detect let alone act on.
Operation Analysis of data from Internet of Things	With more devices connected to the Internet you can get a whole lot more data from these devices -- picture the Nest Learning thermostat and the Power Meter data flowing in and being analyzed together to tell customers how to lower their usage or warn them to lower usage to stave off a blackout. I can almost hear a HAL like voice saying "Dave you lowered your thermostat -- I can't let you do that."
Data warehouse Augmentation/Optimization	While Data warehousing is about aggregating data to look for trends you usually capture data at the transactional level and want to tie it to the customer hence all of those grocery store loyalty cards.
Big Data Efficiency play (break down silos)	Information weekly's example about "A large global financial institution … wanted to move from next-day to same-day balance reporting for its corporate banking customers" again illustrates how identity is the thread that allows you to make meaning of the disparate data.
Big Data Exploration	Since this use case is really about exploring your data to do one of the others it makes more sense to look at how Identity fits in to the other use cases

Conclusion: Scott is right. Without Identity there is no way to act on most of the big data results.

FIM Sync Flow with ScreenShots and Code snippets

Years ago Brad Turner and I created a Flow Chart of FIM data flow with Screenshots and Code snippets. Some of the code examples are funny and it still says ILM rather than FIM. It also doesn't include filter based out bound filter-based sync rules that came with R2. Bearing those things in mind it still provides a good bit of value. Someday I will update it with the latest -- until then enjoy.

FIM Hotfix for PCNS to support 2012 R2 DC's

With the latest hotfix MSFT now supports running PCNS on Windows Server 2012 R2. FIM still should not be installed on Windows Server 2012 R2 (2012 yes, 2008 R2 yes, 2008 yes). Only PCNS can be installed on Windows Server 2012 R2. The hotfix article has a slight error indicating that it is ok to install FIM Sync Service on 2012 R2 if you have installed the hotfix PCNS on 2012 R2 -- not true (the article should get corrected soon). Be warned this update may break ECMA 1 and ECMA 2.0 based MA's. That is they may not run returning "stopped-extension-dll-load" There are workarounds published in the article.

Movie Review of Home -- or how IDM could have saved the day.

Over the weekend I took one of my children to see the new animated film  Home starring Jim Parsons, Rihanna, Steve Martin and Jennifer Lopez. A group of technically superior but very cowardly aliens, called the Boov flee from their implacable enemy, the Gorgs, and decide to take over Earth, relocating all of the primitive natives (us) to Australia. Aside from the political commentary of the entire human race being placed in a reservation, the thing that most struck me was how one of the near disasters could have been averted through solid Identity Management Systems. A hapless and lonely Boov, named "Oh" invited his new neighbors to a "warming of house party." When no one showed, he sought out other acquaintances to invite and sent out an Evite ™ but he accidently did a Send All, which somehow included their implacable enemy. Great hilarity ensues as the evite will take 40 hrs to reach their enemy.

First of all the Boov can't be that superior if they don't have automated Group Management that limits their distribution lists to just those that should be in there, excluding, oh I don't know -- your enemies.

Second, the big brains of the Boov figure out that they can just sign in to "oh's" account and cancel his evite. Their Leader Captain Smek victoriously proclaims "Good thing I made everyone use the same password -- of 'password'." But Oh's password is unique. Scratch that idea. They are clearly lacking the capability for the administrator to reset a password.

Third, Oh finally figures out he needs to cancel his invite, so he attempts to log in to his evite/email account. He fails not quite remembering his password. Fortunately his second attempt succeeds. But had it failed it would have been nice if he could have availed himself of a Self Service Password Reset (SSPR) mechanism.

Finally, their recall email message capability is far better than ours. I mean if the evite took 40 hours to reach the Gorgs, then hour did the recall message reach them instantly? Usually when someone recalls a message it just causes all recipients to read it all the more carefully as they try to find out what was so bad that the sender decided to recall it. In fact if you want to ensure that people read a message attempt to recall it ;)

Portable 2nd Monitor for the Surface Pro 3 ( and TwoMonUSB issues)

As a road warrior, often in different settings, I am interested in a 2nd, portable monitor for my Surface Pro 3. So here was my thought process.

I tried to use TwoMonUSB to make my iPad the second monitor. At first it worked quite well. Great idea, a backup device with some apps I don't have on the surface and I can use it as a second screen. But then my Surface Pro 3 was rebooting randomly during idle times or the screen wouldn't light up after an idle timeout and then I would have to hard boot it. I was getting a bugcheck almost daily, sometimes multiple times a day. So after Refreshing the PC I decided that the $10 app approach wasn't going to work. I am not certain that it was TwoMonUSB but it seems the likely candidate.

But I need a second screen so here are the attributes that are important to me::

Weight: Not weigh me down -> less than 4 lbs

Screen Resolution: Work well with surface -> 1440 x 1080 (if possible)

Power: hopefully it will be able to work without AC available -> less than 5W typical energy consumption

Size: Big enough to see the screen  >12 inches but not so big I can't fit it in my bag <18 inches="" p="">

Price: less than $300 USD

This means that I need a DisplayLink Monitor that weighs less than 4 lbs, use USB for power and transmission of data, is at least 12 inches

Candidates	Weight (lbs)	USB	Screen Res	Power (Max) W	Power Typical (W)	Size	Response Time (ms)	Price (USD)	Comments from reviews
ASUS MB168B+	1.76	3.0	1920x1080		<5 p="">	15.6"	11	299	Works with Surface Pro 3, July 16, 2014 By Glenn Hanner  The Surface Pro 3 will not power the monitor on its own. The USB3 port does not have enough power. From <http://www.amazon.com/MB168B-15-6-Inch-Screen-LED-Lit-Monitor/product-reviews/B00H0FK2A6>
ASUS MB168B	1.76	3.0	1366x768		<5 p="">	15.6"	11	158
Hp S140u	2.76	3.0	1600x900	7	4	14"	8	179
HP U160	3.39	2.0	1366x768	5		15.6"	12	?
AOC E1649FWU	2.34	2.0	1366x768			15.6"	16	90	Works off of Surface Pro 3 battery power One usb http://affinityconsulting.com/technology-tips/315-review-aoc-portable-usb-monitor-for-dual-screen-productivity-on-the-road Andrew Nov 28, 2011 "Uncontrollable brightness… I contacted AOC and they confirmed that you cannot control the brightness" From <http://www.amazon.com/AOC-E1649FWU-USB-Powered-Portable-Monitor/dp/B005SEZR0G>
AOC E1659FWU		3.0	1366x768			15.6"	8	124	Includes carry case, auto pivot
AOC E1759FWU		3.0	1600x900			17.3"	10	200	"The Surface Pro 3 … USB port doesn't put out enough power for the screen. " From <http://www.zdnet.com/article/hands-on-with-the-aoc-e1759fwu-usb-3-monitor/>
Lenovo Think Vision LT1421	2.25	2.0	1366x768			14"			Bulky in back not flat

The ASUS MB168B+ (don't forget the plus) has a great screen resolution but per reviews I won't be able to use it without external power. I was torn -- maybe there would be a way to monkey with brightness settings and get it to work. But the carrying case for it was also not good and lots of folks complain about the stand and it is the most expensive of the items.

I considered the 17 inch AOC E1759FWU that has pretty good screen resolution (1600x900) but again reviews identified the issue with power.

I also considered HP's S140u but I am also concerned that the 7 W maximum power won't work and I could find many reviews, and certainly none where it was paired with the Surface Pro 3.

So I decided that I would try the 16 inch AOC E1659FWU -- I can get a second screen with pretty decent response rate that should (most reviews have said it worked) for my power constraints with the Surface Pro 3. It includes a good case and great stand and is only a little more expensive than its predecessor the AOC E1649FWU.

When all is said and done I may decide that the external power isn't as much of a constraint and go for the ASUS MB168B+ (also need to get a better stand, but that means yet more stuff to carry).

So for now I am looking forward to the arrival of the 16 inch AOC E1659FWU.
----------------------------------------------------------------------------------------------------------------------------

3/17/15 update: Well the AOC E1659FWU is working well, my surface pro 3 can indeed drive the power requirements, which according to the manual is 8 W, whether plugged in or on battery. However, when I plug it in through my Belkin USB hub/Ethernet adapter it won't power up without getting extra power from the USB outlet on my surface charger. Whereas plugging into two of the ports on the Belkin USB hub doesn't do it. (the screen keeps flickering on and off)
Conclusion: If I need to plug in for Ethernet or any other USB device then I must plug in for external power. But the other day I got two good hours of using the surface and the AOC on battery power and still had battery to spare.

Update 8/17/2015
-----------------------------------------
The AOC monitor is very fragile-- despite having it in the provided case inside a padded bag it fell 1.5 feet and the screen cracked. So I have switched back to using my iPad as second display when I am not at the office. But instead of TwoMonUSB I use DuetDisplay.

So how does it compare against my original specs:

Weight: Not weigh me down -- less than 4 lbs -- Check 1.3 lbs

Screen Resolution: Work well with surface -> 1440 x 1080 (if possible) Check I have 3 choices 1024x768, 1532x1152 or 2048x1532. (of course the last one gets hard to read) 

Power: hopefully it will be able to work without AC available -> less than 5W typical energy consumption Check of course  the iPad of course has its own battery

Size: Big enough to see the screen  >12 inches but not so  big I can't fit it in my bag <18 inches="" p="">not quite the iPad 2 has 9.7 inch display

Price: less than $300 USD check -- since I already had it -- otherwise no because it cost over $500 but today you could get a used one for less than $300.

Finally the iPad as you all know is an independent device so if I have some problem with the surface then I can use it to do something.

Escaping an AD Replication Island

On a dark and stormy night an Active Directory upgrade was underway, Windows Server 2003 domain controllers decommissioned, consolidated and replaced with Window Server 2008 R2 servers. Suddenly I got a call from those doing the upgrade, "I can't see some of the new domain controllers on the existing domain controllers, what's wrong?"

A replication island had been created and several domain controllers were trapped on it. Could we rescue them in time?

Normally AD automatically generates the replication topology. But if you turn that off then you must manually create connection objects between domain controllers. Even if that is enabled replication between sites does require site link objects to be created.

With the sites and site links in place and the Knowledge Consistency Checker (KCC) enabled for generating connection objects, the KCC will automatically generate connection objects between Domain Controllers in different sites (those servers are referred to as Bridgehead servers).  By default all site links are transitive. However, this is often turned off if some sites can't connect to others (routing or firewall configurations may be the cause, but it may be legit). However, site links still need to exist.

In this scenario they had the KCC enabled but site link transitivity was off. As domain controllers in several sites were decommissioned leaving some of the new sites with new domain controllers but without direct site links to sites that still had active domain controllers. As I mapped out the new topology I realized that an island had been created -- four of the new sites could talk to each other but not to the other 20 sites.

How to get off of the replication island?

Create Site links to connect the sites to each other! 

But when you create a site link it exists on that domain controller and needs to replicate to the other domain controllers. So how to get it to replicate when you need site links to replicate?   Talk about your chicken and egg problem!

RepAdmin to the rescue! With repadmin /replsingleobj you can force the replication of a single object to any other domain controller even if they aren't replication partners. So after creating the new site links I needed on one of the island domain controllers --  I forced replication among the domain controllers on the island so they all new about the new site link. But the rest of the enterprise still doesn't know so I ran repadmin /replsingleobj NonIslandDC IslandDC "CN=NewIPSITELINK,CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=MyDomain,DC=rootdomain"

Then I forced the replication from that domain controller to its partners and then forced the KCC to generate a new replication topology. The island was bridged.

The domain controllers were rescued!

Follow up #1 on How does Identity Management Impact the Bottom Line? Selling IDM

In my presentation last week at #OCGUS15 The Redmond Summit put on by my friends at OCG, on "How does Identity Management Impact the Bottom Line? Selling IDM" I illustrated how understanding more about Financial statements such as Profit/Loss statements as well as Balance Sheets can be helpful. So here is a link to learn more:

•http://www.investopedia.com/articles/basics/06/financialreporting.asp

Among other things this is helpful to be able to articulate how your projects and programs can impact the bottom line such as how User provisioning/Deprovisioning impacts the Profit and Loss Statement:

Redmond Summit 2015

I am looking forward to presenting in an hour or so on "How Identity Management Impacts the bottom line."

Yesterday I had fun delivering a session on "ADFS vs Password Sync? It depends" This morning Alex Simons of Microsoft revealed a few new things that change some of my advice.
1) Soon Azure AD can do the location restriction by application for SSO. This potentially eliminates a deal breaker for some people
2) You can now run Password Sync and ADFS at the same time.

Both of which make it more likely that you will do Password Sync. The second one makes it more likely that you will run both because Password Sync can be a warm standby for failing over from ADFS.