NIUI: OpenNI / Kinect API for UDK [Beta]

[onethought]

http://www.matthewrobbins.net/images/NIUI%20Logo.png

NIUI is an open-source API that aims to bring Kinect / OpenNI support to the Unreal Development Kit. With the API, I am aiming to remove a lot of the difficulty of working with OpenNI / Kinect and provide an abstracted, gameplay specific layer for depth-camera based games.

All code is provided within the download (including the C++ backend) so you can pick apart how its all implemented. :)

Downloads:
Beta:
NIUI API - Beta v1.0 (http://www.mediafire.com/?kjajd72cwubv0d2)

Alpha:
NIUI API - Alpha v1.1 (http://www.mediafire.com/?g227jkz6tjdxz78)
NIUI API - Alpha v1.0 (http://www.mediafire.com/?p5lxxltv8alnflc)

When you download the alpha, unzip the .rar file and open the "Start Here..." document. This will give instructions in setting up the API.

If you have issues in setting up NIUI, please post a message in this thread or send me a PM. I encourage posting in this thread as the chances are that others may have the same issues, but I will happily respond to PMs.

Before you start playing with the code, I strongly recommend you read the "NITE Algorithms 1.3.pdf" as it will outline some of the fundamentals for working with depth cameras. This document is included in the "Recommended Reading" folder of the download.

If you want a thorough explanation of the API and how to use it, Design3 has a lengthy video tutorial video series on how to use the API. See the tutorial videos section below for a link.

Also, the best way for me to improve the API is through you guys using it and telling me whats good and bad. I'll try to take as much feed-back on board as I can for when I do the beta release of the API.

I will post videos of what I am working on to my channel on YouTube, but I will also maintain a running list here:

Feature Videos:
Kinect / OpenNI to UDK Real-time Bone Rotation (NIUI) (http://www.youtube.com/watch?v=JEDRsA0WsJg)
OpenNI Unreal Interface (NIUI) (http://www.youtube.com/watch?v=CKCkYuKNnCk)

Tutorial Videos:
Design3's NIUI API Video Tutorial Series (http://www.design3.com/udk/intermediate/niui-essentials)
NIUI API Setup Part 1 (http://www.youtube.com/watch?v=DwTLkkvSFaQ)
NIUI API Setup Part 2 (http://www.youtube.com/watch?v=lscuuhorj1k)

[Echo-Leader]

very nice

thankyou looking forward to trying it out, ill let you know if i find any bugs :P

[Elleclouds]

How do I use this? I've followed your instructions to install it and copy the appropriate files but im wondering how would i go about opening up the sample data in UDK? I.e have my kinect recognized by a UDK model.

[Oriam]

I had some problems at the begining with the libraries but I made it work correctly last night. I even opened this account to ask you about my issues but now everything works fine (I had to install the latest unstable openni and nite libraries, which were the ones I had before because I was working with osceleton).

Thanks for your library, it will save me some time in my school project.

[onethought]

How do I use this? I've followed your instructions to install it and copy the appropriate files but im wondering how would i go about opening up the sample data in UDK? I.e have my kinect recognized by a UDK model.

I'm working on a quick start guide at the suggestion of Hacknfly, sorry about the lack of it in the download :(.

I'll do a brief rundown here on how to set up the sample project, as it might help others who may have the same issues. I am assuming here that you have already set up the OpenNI dlls and NIUI.dll correctly (see the Start Here word document).

Step 1:
Firstly, Copy the NIUI folder from the Content/Sample Project folder into your current UDK installs Development/Src folder.

Step 2:
Then go to your UDKGame/Config folder and open up the UDKEngine.ini file.

Step 3:
Do a search for ModEditPackages and change the name after the = to NIUI.
If ModEditPackages doesn't exist in the file, search for [UnrealEd.EditorEngine] and add ModEditPackages=NIUI at the end of the text block.

Step 4:
Now copy the folder NIUI Demo Content from the Content/Unreal Packages folder of the download into your UDKGame/Content folder. This is the assets and packages for the sample project.

You should be right now to view all the content in the editor as well as compile the scripts without any issues.

If you have problems with the above, take a screen shot of the error and give me a description of what happened and I'll do my best to help out.

PS: NIUI was built in the December 2010 version of UDK so I suggest you use that as a minimum. I'll be trying it out in all the versions before the Beta release.


@oriam
Thanks for letting me know about needing the unstable releases. I'll include the needed installers when I do a revision.

[Oriam]

No problem, I´m currently using the february 2011 UDK so that one works too :)

[HackNFly]

Somehow, I forgot to subscribe to this thread, so I've missed the updates. I ran into the same issues. I then followed the new info, installed the latest unstable build, have the NITE directory samples working, but I'm still getting a failure on startup. I'm running Win7 64 bit.



Log: Log file open, 03/05/11 01:59:56
Init: WinSock: version 1.1 (2.2), MaxSocks=32767, MaxUdp=65467
Init: Version: 7876
Init: Epic Internal: 0
Init: Compiled (32-bit): Feb 10 2011 17:20:58
Init: Changelist: 776445
Init: Command line: SG-ExampleGameLevel -norc -resx=1280 -resy=720 -log=test.txt
Init: Base directory: C:\UDK\UDK-2011-02\Binaries\Win32\
DevConfig: GConfig::LoadFile associated file: ..\..\UDKGame\Config\UDKCompat.ini
[0000.29] Init: Computer: SANTIAGO-PC
[0000.29] Init: User: santiago
[0000.29] Init: CPU Page size=4096, Processors=2
[0000.29] Init: High frequency timer resolution =2.727724 MHz
[0000.29] Init: Memory total: Physical=3.0GB Pagefile=6.0GB Virtual=4.0GB
[0000.47] Log: STEAMWORKS initialized 1
[0000.52] Init: WinSock: I am santiago-PC (192.168.1.103:0)
[0000.52] Init: Presizing for 0 objects not considered by GC, pre-allocating 0 bytes.
[0000.52] Init: Object subsystem initialized
[0000.69] Init: OS stats:
[0000.69] Init: Windows 7
[0000.69] Init: RemoteDesktop=0
[0000.69] Init: Memory stats:
[0000.69] Init: Physical: 3070MB
[0000.69] Init: Virtual: 4095MB
[0000.69] Init: PageFile: 6138MB
[0000.69] Init: CPU stats:
[0000.69] Init: MeasuredPerformanceTime: 343.049 (stored result)
[0000.69] Init: Hyperthreaded: 0
[0000.69] Init: NumProcessorsPerCPU: 2
[0000.69] Init: NumLogicalProcessors: 2
[0000.69] Init: NumPhysicalProcessors: 2
[0000.69] Init: MaxSpeed: 2793
[0000.69] Init: CurrentSpeed: 2793
[0000.69] Init: CoresPerProcessor: 2
[0000.69] Init: IsOnBattery: 0
[0000.69] Init: BatteryLevel: -1
[0000.69] Init: Manufacturer: Intel
[0000.69] Init: CPUName: INTEL Pentium-D
[0000.69] Init: L1CacheSize: 16
[0000.69] Init: L2CacheSize: 1024
[0000.69] Init: Architecture: x86
[0000.69] Init: GPU stats:
[0000.69] Init: VendorID: 000010DE
[0000.69] Init: DeviceID: 00000614
[0000.69] Init: DriverVersion: 8.17.12.6099
[0000.69] Init: DeviceName: NVIDIA GeForce 9800 GT
[0000.69] Init: DriverName: nvd3dum.dll
[0000.69] Init: PixelShaderVersion: 3
[0000.69] Init: VertexShaderVersion: 3
[0000.69] Init: VRAMQuantity: 512
[0000.69] Init: DedicatedVRAM: 497
[0000.69] Init: AdapterCount: 2
[0000.69] Init: SupportsHardwareTnL: 1
[0000.69] Init: Machine detected compatibility level: Composite: 3. CPU: 5. GPU: 3.
[0000.69] Init: Previous detected compatibility level: Composite: 3. CPU: 5. GPU: 3.
[0000.83] Log: Shader platform (RHI): PC-D3D-SM3
[0001.24] Log: PhysX GPU Support: DISABLED
[0001.27] Init: Initializing FaceFX...
[0001.27] Init: FaceFX 1.7.3.1 initialized.
[0001.52] Init: XAudio2 using 'Speakers (High Definition Audio Device)' : 2 channels at 44.1 kHz using 16 bits per sample (channel mask 0x3)
[0002.22] DevPhysics: ApexManager: Unable to load legacy Destructible module; loading old Destructible assets may fail.
[0005.27] Init: Finished loading startup packages in 3.37 seconds
[0005.29] Log: 65501 objects as part of root set at end of initial load.
[0005.29] Log: 0 out of 0 bytes used by permanent object pool.
[0005.29] Log: Initializing Engine...
[0005.37] Init: UEngine initialized
[0005.40] Init: XAudio2Device initialized.
[0005.50] Init: Client initialized
[0009.14] Log: Initializing Steamworks
[0009.14] Log: Logged in as 'daThird313'
[0009.33] Log: LoadMap: SG-ExampleGameLevel?Name=Player?Team=255
[0009.48] DevMemory: Memory allocations reported by the OS: 252.79 MB (with 0.00 MB waste)
[0009.48] DevMemory: Virtual memory tracked in the allocators: 95.71 MB (with 87.14 MB used, 2.36 MB slack and 6.21 MB waste)
[0009.55] Log: Game class is 'SantiagoGameInfo'
[0009.58] Log: Primary PhysX scene will be in software.
[0009.58] Log: Creating Primary PhysX Scene.
[0009.58] Log: Bringing World SG-ExampleGameLevel.TheWorld up for play (0) at 2011.03.05-02.00.05
[0009.59] ScriptLog: *************************************************
[0009.59] ScriptLog: NIUI has failed to initilise. Please check the output log for details.
[0009.59] Log: Bringing up level for play took: 0.027688
[0009.60] ScriptWarning: Accessed None 'TPawn'
SimpleCamera SG-ExampleGameLevel.TheWorld:PersistentLevel.SimpleCa mera_0
Function NIUI.SimpleCamera:UpdateViewTarget:0344
[0009.60] ScriptWarning: Accessed None 'TPawn'
SimpleCamera SG-ExampleGameLevel.TheWorld:PersistentLevel.SimpleCa mera_0
Function NIUI.SimpleCamera:UpdateViewTarget:0386
[0009.60] ScriptWarning: Accessed None 'TPawn'
SimpleCamera SG-ExampleGameLevel.TheWorld:PersistentLevel.SimpleCa mera_0
Function NIUI.SimpleCamera:UpdateViewTarget:03BA
[0009.60] ScriptWarning: Accessed None
SimpleCamera SG-ExampleGameLevel.TheWorld:PersistentLevel.SimpleCa mera_0
Function NIUI.SimpleCamera:UpdateViewTarget:03CE
[0009.60] ScriptWarning: Accessed None 'TPawn'
SimpleCamera SG-ExampleGameLevel.TheWorld:PersistentLevel.SimpleCa mera_0
Function NIUI.SimpleCamera:UpdateViewTarget:04DE
[0009.60] ScriptWarning: Accessed None 'TPawn'
SimpleCamera SG-ExampleGameLevel.TheWorld:PersistentLevel.SimpleCa mera_0
Function NIUI.SimpleCamera:UpdateViewTarget:060E
[0009.62] ScriptLog: NIUI_SkelControlSingleBone_0 targeting joint: NIUI_JOINT_TORSO
[0009.62] ScriptLog: NIUI_SkelControlSingleBone_2 targeting joint: NIUI_JOINT_LEFT_SHOULDER
[0009.62] ScriptLog: NIUI_SkelControlSingleBone_3 targeting joint: NIUI_JOINT_LEFT_ELBOW
[0009.62] ScriptLog: NIUI_SkelControlSingleBone_8 targeting joint: NIUI_JOINT_RIGHT_SHOULDER
[0009.62] ScriptLog: NIUI_SkelControlSingleBone_9 targeting joint: NIUI_JOINT_RIGHT_ELBOW
[0009.62] ScriptLog: NIUI_SkelControlSingleBone_1 targeting joint: NIUI_JOINT_NECK
[0009.62] ScriptLog: NIUI_SkelControlSingleBone_5 targeting joint: NIUI_JOINT_LEFT_HIP
[0009.62] ScriptLog: NIUI_SkelControlSingleBone_6 targeting joint: NIUI_JOINT_LEFT_KNEE
[0009.62] ScriptLog: NIUI_SkelControlSingleBone_11 targeting joint: NIUI_JOINT_RIGHT_HIP
[0009.62] ScriptLog: NIUI_SkelControlSingleBone_12 targeting joint: NIUI_JOINT_RIGHT_KNEE
[0009.62] Error: StartLocalVoiceProcessing(): Device is currently owned by another user
[0009.62] Log: ########### Finished loading level: 0.291545 seconds
[0009.66] Init: Game engine initialized
[0009.66] Log: Initializing Engine Completed
[0012.30] Log: >>>>>>>>>>>>>> Initial startup: 12.30s <<<<<<<<<<<<<<<
[0012.31] ScriptLog: NIUI Dependency found of class NIUI_SamplePlayerController. Notifying of Core Creation.
[0012.31] ScriptLog: #### ThirdPerson
[0055.18] Log: Assembled 353 auto-complete commands, manual: 77, exec: 259, kismet: 0
[0087.18] ScriptWarning: bRestartLevel && !server, abort from RestartPlayer NM_Standalone
SantiagoGameInfo SG-ExampleGameLevel.TheWorld:PersistentLevel.Santiago GameInfo_0
Function Engine.GameInfo:RestartPlayer:00BB
[0101.77] ScriptWarning: Invalid user index (255) specified for ClearReadProfileSettingsCompleteDelegate()
OnlineSubsystemSteamworks Transient.OnlineSubsystemSteamworks_0
Function OnlineSubsystemSteamworks.OnlineSubsystemSteamwork s:ClearReadProfileSettingsCompleteDelegate:00FE
[0101.77] Log: All Windows Closed
[0101.77] Log: appRequestExit(0)
[0101.85] Exit: Preparing to exit.
[0102.13] Exit: Game engine shut down
[0102.15] Exit: Windows client shut down
[0102.16] Exit: XAudio2 Device shut down.
[0102.29] Exit: Object subsystem successfully closed.
[0102.31] Log: Shutting down FaceFX...
[0102.31] Log: FaceFX shutdown.
[0102.37] Exit: Exiting.
[0102.38] Log: Log file closed, 03/05/11 02:01:38


My level is just a big cube, with a playerstart in the center, my gaminfo says SantiagoGameInfo, but i just copied and pasted the NIUI gameinfo file content inside of it. The level starts up with the proto man, and he can move around and run, but no kinect. Any ideas?

[onethought]

Does any message boxes appear?
Could you post the code for your gameinfo, player controller and pawn?
Also, double check the NIUIConfig is within the usercode folder.

Lastly, download the dll from here (http://www.matthewrobbins.net/Downloads/NIUI.dll) and copy it over the existing NIUI.dll in the usercode folder, just to make sure its the right dll.

[HackNFly]

Have you updated things since your original post? Because that dll did at least get the camera to connect, however as soon as I step into the frame it says player detected, then I hold up my hands then I get a bunch of missing information in the log. Oddly enough I can't find where in the log it says player detected like it did on the live log

Here is how I'm launching the game, in case that messes with how the log it output (Had to get use to running it from cmd prompt since new frontend doesn't work the same)

C:\UDK\UDK-2011-02\Binaries\UDK.exe SG-ExampleGameLevel -norc -resx=1280 -resy=720 -log

And here is the relevant error from the log:


Log: Log file open, 03/05/11 02:35:48
Init: WinSock: version 1.1 (2.2), MaxSocks=32767, MaxUdp=65467
Init: Version: 7876
Init: Epic Internal: 0
Init: Compiled (32-bit): Feb 10 2011 17:20:58
Init: Changelist: 776445
Init: Command line: SG-ExampleGameLevel -norc -resx=1280 -resy=720 -log
Init: Base directory: C:\UDK\UDK-2011-02\Binaries\Win32\
DevConfig: GConfig::LoadFile associated file: ..\..\UDKGame\Config\UDKCompat.ini
[0000.25] Init: Computer: SANTIAGO-PC
[0000.25] Init: User: santiago
[0000.25] Init: CPU Page size=4096, Processors=2
[0000.25] Init: High frequency timer resolution =2.727724 MHz
[0000.25] Init: Memory total: Physical=3.0GB Pagefile=6.0GB Virtual=4.0GB
[0000.42] Log: STEAMWORKS initialized 1
[0000.48] Init: WinSock: I am santiago-PC (192.168.1.103:0)
[0000.48] Init: Presizing for 0 objects not considered by GC, pre-allocating 0 bytes.
[0000.49] Init: Object subsystem initialized
[0000.72] Init: OS stats:
[0000.72] Init: Windows 7
[0000.72] Init: RemoteDesktop=0
[0000.72] Init: Memory stats:
[0000.72] Init: Physical: 3070MB
[0000.72] Init: Virtual: 4095MB
[0000.73] Init: PageFile: 6138MB
[0000.73] Init: CPU stats:
[0000.73] Init: MeasuredPerformanceTime: 343.049 (stored result)
[0000.73] Init: Hyperthreaded: 0
[0000.74] Init: NumProcessorsPerCPU: 2
[0000.74] Init: NumLogicalProcessors: 2
[0000.74] Init: NumPhysicalProcessors: 2
[0000.74] Init: MaxSpeed: 2793
[0000.74] Init: CurrentSpeed: 2793
[0000.74] Init: CoresPerProcessor: 2
[0000.74] Init: IsOnBattery: 0
[0000.74] Init: BatteryLevel: -1
[0000.74] Init: Manufacturer: Intel
[0000.75] Init: CPUName: INTEL Pentium-D
[0000.75] Init: L1CacheSize: 16
[0000.75] Init: L2CacheSize: 1024
[0000.75] Init: Architecture: x86
[0000.75] Init: GPU stats:
[0000.75] Init: VendorID: 000010DE
[0000.75] Init: DeviceID: 00000614
[0000.76] Init: DriverVersion: 8.17.12.6099
[0000.76] Init: DeviceName: NVIDIA GeForce 9800 GT
[0000.76] Init: DriverName: nvd3dum.dll
[0000.76] Init: PixelShaderVersion: 3
[0000.76] Init: VertexShaderVersion: 3
[0000.76] Init: VRAMQuantity: 512
[0000.77] Init: DedicatedVRAM: 497
[0000.77] Init: AdapterCount: 2
[0000.77] Init: SupportsHardwareTnL: 1
[0000.77] Init: Machine detected compatibility level: Composite: 3. CPU: 5. GPU: 3.
[0000.77] Init: Previous detected compatibility level: Composite: 3. CPU: 5. GPU: 3.
[0000.94] Log: Shader platform (RHI): PC-D3D-SM3
[0001.50] Log: PhysX GPU Support: DISABLED
[0001.51] Init: Initializing FaceFX...
[0001.52] Init: FaceFX 1.7.3.1 initialized.
[0001.76] Init: XAudio2 using 'Speakers (High Definition Audio Device)' : 2 channels at 44.1 kHz using 16 bits per sample (channel mask 0x3)
[0002.72] DevPhysics: ApexManager: Unable to load legacy Destructible module; loading old Destructible assets may fail.
[0006.12] Init: Finished loading startup packages in 3.88 seconds
[0006.14] Log: 65501 objects as part of root set at end of initial load.
[0006.14] Log: 0 out of 0 bytes used by permanent object pool.
[0006.15] Log: Initializing Engine...
[0006.25] Init: UEngine initialized
[0006.31] Init: XAudio2Device initialized.
[0006.57] Init: Client initialized
[0010.93] Log: Initializing Steamworks
[0010.93] Log: Logged in as 'daThird313'
[0011.52] Log: LoadMap: SG-ExampleGameLevel?Name=Player?Team=255
[0011.76] DevMemory: Memory allocations reported by the OS: 253.65 MB (with 0.00 MB waste)
[0011.76] DevMemory: Virtual memory tracked in the allocators: 95.64 MB (with 87.14 MB used, 2.30 MB slack and 6.21 MB waste)
[0011.84] Log: Game class is 'SantiagoGameInfo'
[0011.95] Log: Primary PhysX scene will be in software.
[0011.96] Log: Creating Primary PhysX Scene.
[0011.96] Log: Bringing World SG-ExampleGameLevel.TheWorld up for play (0) at 2011.03.05-02.36.00
[0011.96] ScriptLog: *************************************************
[0018.61] ScriptLog: *************************************************
[0018.62] Log: Bringing up level for play took: 6.764176
[0018.63] ScriptWarning: Accessed None 'TPawn'
SimpleCamera SG-ExampleGameLevel.TheWorld:PersistentLevel.SimpleCa mera_0
Function NIUI.SimpleCamera:UpdateViewTarget:0344
[0018.63] ScriptWarning: Accessed None 'TPawn'
SimpleCamera SG-ExampleGameLevel.TheWorld:PersistentLevel.SimpleCa mera_0
Function NIUI.SimpleCamera:UpdateViewTarget:0386
[0018.63] ScriptWarning: Accessed None 'TPawn'
SimpleCamera SG-ExampleGameLevel.TheWorld:PersistentLevel.SimpleCa mera_0
Function NIUI.SimpleCamera:UpdateViewTarget:03BA
[0018.63] ScriptWarning: Accessed None
SimpleCamera SG-ExampleGameLevel.TheWorld:PersistentLevel.SimpleCa mera_0
Function NIUI.SimpleCamera:UpdateViewTarget:03CE
[0018.64] ScriptWarning: Accessed None 'TPawn'
SimpleCamera SG-ExampleGameLevel.TheWorld:PersistentLevel.SimpleCa mera_0
Function NIUI.SimpleCamera:UpdateViewTarget:04DE
[0018.64] ScriptWarning: Accessed None 'TPawn'
SimpleCamera SG-ExampleGameLevel.TheWorld:PersistentLevel.SimpleCa mera_0
Function NIUI.SimpleCamera:UpdateViewTarget:060E
[0018.67] ScriptLog: NIUI_SkelControlSingleBone_0 targeting joint: NIUI_JOINT_TORSO
[0018.67] ScriptLog: NIUI_SkelControlSingleBone_2 targeting joint: NIUI_JOINT_LEFT_SHOULDER
[0018.67] ScriptLog: NIUI_SkelControlSingleBone_3 targeting joint: NIUI_JOINT_LEFT_ELBOW
[0018.67] ScriptLog: NIUI_SkelControlSingleBone_8 targeting joint: NIUI_JOINT_RIGHT_SHOULDER
[0018.67] ScriptLog: NIUI_SkelControlSingleBone_9 targeting joint: NIUI_JOINT_RIGHT_ELBOW
[0018.68] ScriptLog: NIUI_SkelControlSingleBone_1 targeting joint: NIUI_JOINT_NECK
[0018.68] ScriptLog: NIUI_SkelControlSingleBone_5 targeting joint: NIUI_JOINT_LEFT_HIP
[0018.68] ScriptLog: NIUI_SkelControlSingleBone_6 targeting joint: NIUI_JOINT_LEFT_KNEE
[0018.68] ScriptLog: NIUI_SkelControlSingleBone_11 targeting joint: NIUI_JOINT_RIGHT_HIP
[0018.69] ScriptLog: NIUI_SkelControlSingleBone_12 targeting joint: NIUI_JOINT_RIGHT_KNEE
[0018.70] Error: StartLocalVoiceProcessing(): Device is currently owned by another user
[0018.70] Log: ########### Finished loading level: 7.174536 seconds
[0018.78] Init: Game engine initialized
[0018.78] Log: Initializing Engine Completed
[0018.92] Log: >>>>>>>>>>>>>> Initial startup: 18.92s <<<<<<<<<<<<<<<
[0018.95] ScriptLog: NIUI Dependency found of class NIUI_SamplePlayerController. Notifying of Core Creation.
[0018.95] ScriptLog: #### ThirdPerson
[0047.38] ScriptWarning: Accessed None 'MyPawn'
NIUI_SamplePlayerController SG-ExampleGameLevel.TheWorld:PersistentLevel.NIUI_Sam plePlayerController_0
Function SantiagoGame.NIUI_SamplePlayerController:OnNIUIEve nt:0189
[0049.59] ScriptWarning: Accessed None 'Pawn'
NIUI_SamplePlayerController SG-ExampleGameLevel.TheWorld:PersistentLevel.NIUI_Sam plePlayerController_0
Function SantiagoGame.NIUI_SamplePlayerController:OnNIUIEve nt:00D8


and it repeats, obviously its complaining about MyPawn...

Oh yeah, and as far as my classes, they are the same as everything in the NIUI example folder except that the initial gameinfo class has a one word subtitution

class SantiagoGameInfo extends FrameworkGame;

[onethought]

The live log will be being printed out through native code so it won't appear in the UDK log. Can you do a screen shot of the console log?

I'll do an update tomorrow to dump out the information to a text file and fix the dll issue.

[pesdelisi]

Hi my friend. Firstly thanks for your work. And for the Turkish Developers i'm translated document to Turkish. You can release package with that. When you release new version, please send pm to me, i'll translate new one.

Thanks
Atahan Bozkurt


UDK ile Kinect'i Kullanmak

Buradan alpha sürümünü indirebilirsiniz. Bu yazı içerisindeki dökümanın Türkçe'ye çevirisidir.

http://www.mediafire.com/?p5lxxltv8alnflc

NIUI'yi Ayarlamak

1. Adım:
Öncelikle aşağıdaki bileşenlerie indirin ve kurun.
OpenNI Kütüphanesi:
Link: http://www.openni.org/downloadfiles/openni-binaries/21-stable
Windows için son kararlı sürümü indirin.
NITE MiddleWare:
Link: http://www.openni.org/downloadfiles/openni-compliant-middleware-binaries/34-stable
Windows için son kararlı sürümü indirin.
Bunu indirebilmek için key'e ihtiyacınız olacak. PrimeSense tarafından sağlanan key'i kullanabilirsiniz: 0KOIk2JeIBYClPWVnMoRKn5cdY4=
OPEN-NI WEBSİTESİN'DEN DEPTH-CAMERA SÜRÜCÜSÜNÜ İNDİRİP & KURMAYINIZ.
2. Adım:
Söylediğim gibi OpenNI websitesindeki sürücü Kinect'e uygun değil, yani PrimeSense Depth-Camera sürücüsünü kullanmayın, işe yaramayacaktır.
Sürücü Extras klasöründeki SensorKinect'dir.
Bu PrimeSense sürücüsünün Kinect'e göre düzenlenmiş halidir.
NOTE: NITE örnekleri yüklesenizde çalışmayacaktır (en azından benim deneyimlerim bu şekilde.).
3.Adım
Yukarıdakileri kurduktan sonra "Required dlls" adlı klasöre girin ve "To be copied into Win32" klasörünün içindekileri Win32'ye atın. Daha sonra "To be copied into Win32" klasöründeki dll'leri UDKSürümünüz /Binaries/Win32'ye atın. Ve bütün .dll'lerin UDK.exe ile aynı klasörde olduğundan emin olun.
4.Adım
NIUI.dll'yi UserCode klasörünüze kopyalayın. NIUI.dll Required Dlls içinde bulunmakta. Ayrıca UserCode klasörünüz UDKSürümünüz \ Binaires \Win32 klasörünün içinde olması gerekiyor.
5. Adım
"Content/UnrealScript API" klasöründekileri sizin proje dizininize atın. İçindeki 5 dosyayı kendi projenide nasıl kullanılacağından fikir sahibi olmak için SampleProject'in PlayerController, GameInfo ve Pawn classlarına bakın.
6.Adım
Content içindeki NIUIConfig.xml'i bulun ve UserCode klasörünüze koyun. Bu OpenNI için kullanılacak ayarlardır.
7.Adım

UnrealPackages içindeki "NIUI Demo Content"'i UDKDiziniz/UDKGame/Content içine koyun. Bu demo için gereklidir.
8.Adım
GameInfo clasasındaki (NIUI_Main) içindeki ayarları ve NIUI_samplePlayerController'daki olay oluşturmayı ve pawn'ınızı nasıl oluşturup kullanacağınız hakındaki NIUI_SamplePawn'ı inceleyin.

Ayrıca, Extras klasörü içindeki Anim-Tree dökümasyonuna bakarak kemik sisteminizi nasıl ilişkilendirebileceğinize bakabilirsiniz. Ayrıca mesh'inize assign edeceğiniz, OpenNI ile uyumlu TPose yani SkeletonController'in rotasyonu önbelleğe alması için gereken bir animasyona ihtiyacınız var. Örnek olarak içerik paketleri ve NIUI_SamplePawn.uc dosyasına bakabilirsiniz.
Ayrıca kesinlikle "NITE Algorithms 1.3.pdf"yi okumanızı öneririm. Burada anahtar konseptler ve OpenNI ile çalışırken gerekli olan temel bilgiye ulaşabilirsiniz.
9.Adım
Sonuç olarak çalışmam için zaman ayırdığınız için teşekkür ederim. VE unutmayın ki bu API'nin alpha sürümüdür yani mükemmel fonksiyonellik beklemeyin. Aklımda beta ve final versiyon için tasarladığım fikirler var.
Ayrıca OpenNI için bir installer oluşturmaya uğraşmadım. Çünkü bu yol ile problemler çıkabilir. Beta versiyonu için bir tutorial üzerinde çalışıyorum. Ayrıca bir projeniz var ise OpenNI logosunu koymayı unutmayın.
Eğer bug, hata veya öneriniz var ise Unreal Forumlarından bana ulaşabilirsiniz. Kullanıcı adım OneThought'dur.

Çeviri: Atahan Bozkurt

[Oriam]

I forgot to mention this the last time but I noticed the xml path for the config file is hardcoded to C:/NIUIConfig.xml, maybe that´s why you can't initialize kinect. I think there may be a bug in here (I haven´t checked the code in detail though, and I apologize if I´m talking nonsense)

int InitiliseNIUI(int useXMLInit, int requestedNumberOfPlayers)
{
status = g_core.Initilise("C:\\NIUIConfig.xml", requestedNumberOfPlayers, useXMLInit== 1);
....
}

....

if (useXMLInit)
{
LOG("Core has been requested to initilise from an xml file.", LOG_Init);
result = m_openNIInfo->context.InitFromXmlFile(xmlPath);
....
}
else
{
LOG("Core has been requested to initilise as default.", LOG_Init);
printf("default init\n");
result = m_openNIInfo->context.InitFromXmlFile(xmlPath);
....
}

The code initializes from the xml file in both the if and the else

Maybe you can fix your problem if you put the config file in C: (I did it but since I moved more things, including the install of the unstable builds, I don´t know if that helped at all)

[HackNFly]

I tried puting the file in C:\ and it worked, but afterwards, it worked without it. By it worked I mean the initialization error is gone, but the missing mypawn issue is still there and nothing is working, I can see the proto human guy on startup though. It goes through the NUI initialization and says it is successful, but I get the same repeating error about missing pawn (see bottom of my previous post). I'm wondering if I set up my level improperly to work with it.

[onethought]

I tried puting the file in C:\ and it worked, but afterwards, it worked without it. By it worked I mean the initialization error is gone, but the missing mypawn issue is still there and nothing is working, I can see the proto human guy on startup though. It goes through the NUI initialization and says it is successful, but I get the same repeating error about missing pawn (see bottom of my previous post). I'm wondering if I set up my level improperly to work with it.

Hey guys,

Really sorry about that bad file reference issue, I should have spotted that before I uploaded it as it essentially breaks the entire API. I've already fixed it on my end and am currently tidying things up. I should be starting to upload a revision that will fix the problem shortly.

Ill do another post when its available.

Again, apologies about the bad file reference :(

[HackNFly]

Sorry, I guess I wasn't specific, but the reference wasn't the error. I'm not sure what was, the NIUI seems to be initializing, but I keep getting the pawn error. I deleted the file from c:\ and it made no difference, the code made it seem like it actually wasn't being used there.

[onethought]

Bump post notifying the release of NIUI API Alpha v1.1.
See first post for download link.

[HackNFly]

First great job with the KinectInstaller, first one I've seen around. My Kinect was already working, but I also tried uninstalling all my drivers and reinstalling using the installer you built, it worked great.
---

I am still having some problems though...

Maybe this is lack of experience talking, but aren't modifications suppose to be made to the Default***.ini files and not the UDK ones, I thought the UDK ones get rewritten everytime you perform a full recompile.

Also, I'm having a hard time figuring out how you are exactly suppose to run the demo. I followed the quick install and copy and pasted the files. The new Frontend had me stumped for a while. I ended up cloning DM-deck, renamed it NIUI and under Maps to Cook clicked NIUI_Content.udk, although it seemed like the other map would have been the right one to choose, although it didn't show up. clicking start on the frontend ended up with the default Unreal Game Menu coming up, I selected NIUI under maps and it crashed when I tried to start.

I also tried running it by making a shortcut to UDK.exe with the following information in the target box:
C:\UDK\UDK-2010-12\Binaries\UDK.exe NIUI_Content -norc -resx=1280 -resy=720 -log

I also figured that the map directory may have been in the wrong place, so I tried moving that and using the NIUITestbed.udk to the Content/Maps folder and choosing that one, but I still don't get the right game type to show up, any suggestions?

[onethought]

I haven't had much experience with frontend so I suggest that you start the demo through nFringe and Visaul Studio. The code project file should already be set up to start up the demo.

As for the NIUI_Content.udk, thats one of the packages that contains some samples. After checking it out in the editor, I think it might be broken so just ignore it. The main package that contains all the demo content is the "NIUI_Demo_Content.upk" package.

You will need to use the NIUITestbed.udk map as the testbed map.

Alternatively, try creating a new shortcut and pasting the following as the shortcut location (don't browse to the UDK.exe, just paste this in):

"C:\UDK\UDK-2010-12\Binaries\Win32\UDK.exe" NIUITestBed?game=NIUI.NIUI_Main -vadebug -useunpublished -windowed -resx=1280 -resy=960 -log -ConsolePosX=0 -ConsolePosY=0

Make sure you compile all the scripts in Front-End before doing the above step.

I'll add these steps to the Quick-Start tutorial in the next revision.

EDIT:
Just to clarify, I didn't write that awesome Kinect Installer program. You can find it here (http://babaandthepigman.wordpress.com/2011/01/26/openni-kinect-getting-set-up-on-windows/)

[HackNFly]

Thanks for the update, that clarifies a few things, I tried it on my desktop and had some luck. I've wanted to use NFringe, I'm a fan of Visual Studio, but the tremendous price tag puts me off. (In case we wanted to go commercial).



Log: Log file open, 03/10/11 11:56:09
Init: WinSock: version 1.1 (2.2), MaxSocks=32767, MaxUdp=65467
Init: Version: 7637
Init: Epic Internal: 0
Init: Compiled (32-bit): Dec 11 2010 09:29:40
Init: Changelist: 720882
Init: Command line: NIUITestBed?game=NIUI.NIUI_Main -vadebug -useunpublished -windowed -resx=1280 -resy=960 -log -ConsolePosX=0 -ConsolePosY=0
Init: Base directory: C:\UDK\UDK-2010-12\Binaries\Win32\
DevConfig: GConfig::LoadFile associated file: ..\..\UDKGame\Config\UDKCompat.ini
[0000.32] Init: Computer: SANTIAGO-PC
[0000.32] Init: User: santiago
[0000.32] Init: CPU Page size=4096, Processors=2
[0000.32] Init: High frequency timer resolution =2.727656 MHz
[0000.32] Init: Memory total: Physical=3.0GB Pagefile=6.0GB Virtual=4.0GB
[0000.51] Log: STEAMWORKS initialized 1
[0000.56] Init: WinSock: I am santiago-PC (192.168.1.103:0)
[0000.56] Init: Presizing for 0 objects not considered by GC, pre-allocating 0 bytes.
[0000.56] Init: Object subsystem initialized
[0000.74] Init: OS stats:
[0000.74] Init: Windows 7
[0000.74] Init: RemoteDesktop=0
[0000.74] Init: Memory stats:
[0000.74] Init: Physical: 3070MB
[0000.74] Init: Virtual: 4095MB
[0000.74] Init: PageFile: 6138MB
[0000.74] Init: CPU stats:
[0000.74] Init: MeasuredPerformanceTime: 174.485 (stored result)
[0000.74] Init: Hyperthreaded: 0
[0000.74] Init: NumProcessorsPerCPU: 2
[0000.74] Init: NumLogicalProcessors: 2
[0000.74] Init: NumPhysicalProcessors: 2
[0000.75] Init: MaxSpeed: 2793
[0000.75] Init: CurrentSpeed: 2793
[0000.75] Init: CoresPerProcessor: 2
[0000.75] Init: IsOnBattery: 0
[0000.75] Init: BatteryLevel: -1
[0000.75] Init: Manufacturer: Intel
[0000.76] Init: CPUName: INTEL Pentium-D
[0000.76] Init: L1CacheSize: 16
[0000.76] Init: L2CacheSize: 1024
[0000.76] Init: Architecture: x86
[0000.76] Init: GPU stats:
[0000.76] Init: VendorID: 000010DE
[0000.76] Init: DeviceID: 00000614
[0000.77] Init: DriverVersion: 8.17.12.6099
[0000.77] Init: DeviceName: NVIDIA GeForce 9800 GT
[0000.77] Init: DriverName: nvd3dum.dll
[0000.77] Init: PixelShaderVersion: 3
[0000.77] Init: VertexShaderVersion: 3
[0000.77] Init: VRAMQuantity: 512
[0000.78] Init: DedicatedVRAM: 497
[0000.78] Init: AdapterCount: 2
[0000.78] Init: SupportsHardwareTnL: 1
[0000.78] Init: Machine detected compatibility level: Composite: 3. CPU: 5. GPU: 3.
[0000.79] Init: Previous detected compatibility level: Composite: 4. CPU: 5. GPU: 4.
[0000.97] Log: Shader platform (RHI): PC-D3D-SM3
[0002.06] Log: PhysX GPU Support: DISABLED
[0002.07] Init: Initializing FaceFX...
[0002.07] Init: FaceFX 1.7.3.1 initialized.
[0005.87] Init: Finished loading startup packages in 3.36 seconds
[0005.89] Log: 64699 objects as part of root set at end of initial load.
[0005.89] Log: 0 out of 0 bytes used by permanent object pool.
[0005.89] Log: Initializing Engine...
[0005.97] Init: UEngine initialized
[0005.97] Log: Attaching script debugger (Visual Studio interface)
[0005.97] Init: UnrealScript Debugger Core Initialized.
[0005.98] Init: XAudio2 using 'Speakers (High Definition Audio Device)' : 2 channels at 44.1 kHz using 16 bits per sample (channel mask 0x3)
[0006.04] Init: XAudio2Device initialized.
[0006.14] Init: Client initialized
[0009.50] Log: Initializing Steamworks
[0009.50] Log: Logged in as 'daThird313'
[0009.80] Log: LoadMap: NIUITestBed?Name=Player?Team=255?game=NIUI.NIUI_Ma in
[0009.92] DevMemory: Virtual memory allocation size: 90.83 MByte (95244288 Bytes)
[0009.98] Log: Game class is 'NIUI_Main'
[0010.01] Log: Primary PhysX scene will be in software.
[0010.01] Log: Creating Primary PhysX Scene.
[0010.02] Log: Bringing World NIUITestBed.TheWorld up for play (0) at 2011.03.10-11.56.19
[0010.02] ScriptLog: *************************************************
[0016.27] ScriptLog: *************************************************
[0016.27] Log: Bringing up level for play took: 6.275268
[0016.27] ScriptWarning: Accessed None 'TPawn'
SimpleCamera NIUITestBed.TheWorld:PersistentLevel.SimpleCamera_ 0
Function NIUI.SimpleCamera:UpdateViewTarget:0344
[0016.27] ScriptWarning: Accessed None 'TPawn'
SimpleCamera NIUITestBed.TheWorld:PersistentLevel.SimpleCamera_ 0
Function NIUI.SimpleCamera:UpdateViewTarget:0386
[0016.28] ScriptWarning: Accessed None 'TPawn'
SimpleCamera NIUITestBed.TheWorld:PersistentLevel.SimpleCamera_ 0
Function NIUI.SimpleCamera:UpdateViewTarget:03BA
[0016.28] ScriptWarning: Accessed None
SimpleCamera NIUITestBed.TheWorld:PersistentLevel.SimpleCamera_ 0
Function NIUI.SimpleCamera:UpdateViewTarget:03CE
[0016.28] ScriptWarning: Accessed None 'TPawn'
SimpleCamera NIUITestBed.TheWorld:PersistentLevel.SimpleCamera_ 0
Function NIUI.SimpleCamera:UpdateViewTarget:04DE
[0016.28] ScriptWarning: Accessed None 'TPawn'
SimpleCamera NIUITestBed.TheWorld:PersistentLevel.SimpleCamera_ 0
Function NIUI.SimpleCamera:UpdateViewTarget:060E
[0016.30] ScriptLog: NIUI_SkelControlSingleBone_0 targeting joint: NIUI_JOINT_TORSO
[0016.30] ScriptLog: NIUI_SkelControlSingleBone_2 targeting joint: NIUI_JOINT_LEFT_SHOULDER
[0016.30] ScriptLog: NIUI_SkelControlSingleBone_3 targeting joint: NIUI_JOINT_LEFT_ELBOW
[0016.30] ScriptLog: NIUI_SkelControlSingleBone_8 targeting joint: NIUI_JOINT_RIGHT_SHOULDER
[0016.30] ScriptLog: NIUI_SkelControlSingleBone_9 targeting joint: NIUI_JOINT_RIGHT_ELBOW
[0016.30] ScriptLog: NIUI_SkelControlSingleBone_1 targeting joint: NIUI_JOINT_NECK
[0016.30] ScriptLog: NIUI_SkelControlSingleBone_5 targeting joint: NIUI_JOINT_LEFT_HIP
[0016.31] ScriptLog: NIUI_SkelControlSingleBone_6 targeting joint: NIUI_JOINT_LEFT_KNEE
[0016.31] ScriptLog: NIUI_SkelControlSingleBone_11 targeting joint: NIUI_JOINT_RIGHT_HIP
[0016.31] ScriptLog: NIUI_SkelControlSingleBone_12 targeting joint: NIUI_JOINT_RIGHT_KNEE
[0016.32] Error: StartLocalVoiceProcessing(): Device is currently owned by another user
[0016.32] Log: ########### Finished loading level: 6.524484 seconds
[0016.37] Init: Game engine initialized
[0016.37] Log: Initializing Engine Completed
[0016.39] Log: >>>>>>>>>>>>>> Initial startup: 16.39s <<<<<<<<<<<<<<<
[0016.41] ScriptLog: NIUI Dependency found of class NIUI_SamplePlayerController. Notifying of Core Creation.
[0016.41] ScriptLog: #### ThirdPerson
[0016.41] ScriptLog: NIUI Dependency found of class NIUI_SamplePawn. Notifying of Core Creation.
[0098.85] ScriptWarning: Invalid user index (255) specified for ClearReadProfileSettingsCompleteDelegate()
OnlineSubsystemSteamworks Transient.OnlineSubsystemSteamworks_0
Function OnlineSubsystemSteamworks.OnlineSubsystemSteamwork s:ClearReadProfileSettingsCompleteDelegate:00FE
[0098.86] Log: All Windows Closed
[0098.86] Log: appRequestExit(0)
[0098.94] Exit: Preparing to exit.
[0099.24] Exit: Game engine shut down
[0099.26] Exit: Windows client shut down
[0099.30] Exit: XAudio2 Device shut down.
[0099.46] Exit: Object subsystem successfully closed.
[0099.47] Log: Shutting down FaceFX...
[0099.47] Log: FaceFX shutdown.
[0099.53] Exit: Exiting.
[0099.53] Log: Log file closed, 03/10/11 11:57:49

Which is interesting, in that i didn't notice the invalid user index, however, I think it only shows up on cleanup, if no user was found while running the program. So I still have the accessed none TPawn warnings, but it does detect when the user enters the frame, it just never "locks in" on the user when I raise my hands in the Psi pose. This is running on my Core 2 Duo, which is a lower end machine. I had the same issue when trying it on my own code I developed, I could never get it to lock in while running the UDK, i figured it was an issue of not enough free clock cycles. I'm currently moving everything over to our groups core i7 laptop and retrying. It looks like I'm a lot closer to getting it working.

Also, the Frontend looks for maps under UDKGame/Content/Maps which may or may not be different from NFringe.

[onethought]

@HacknFly

Are you using the sample code for this? If so, then it should work.
Can I see the output log window as a screen shot?

If this is your own code, am I able to look at how your handling the events in your player controller?

[HackNFly]

This is the demo version, with a fresh install of UDK-2010-12, and I followed the quick install to the T. For all intensive purposes it looks likes its running, I really think my CPU just isn't fast enough. The only reason I was able to get the Kinect to run with UDK at all when I did my code was because I locked in on myself before UDK launched. However, I'm transferring over the UDK folder to the laptop right now to test out that theory. Here is a screen capture of how it currently looks like.

http://img857.imageshack.us/img857/3299/niuidump.png

[HackNFly]

Just wanted to update that I tried it on the laptop and got the same issue, it says user detected, but doesn't do anything when I hold the psi position. I can do the psi position fine too, because it works on the players demo from the NITE sample directory just fine.

[FIXED]

SUCCESSSS!!!!
I took my own advice and tried the NITE players sample. It appears that in all the moving and reconfiguring of the Kinect camera, that it was now pointing down slightly, so my normal position in my room with the PSI pose was not being picked up. I quickly ran my code which resets the camera position to zero. And had quick success, even on my crappy old machine!!!!

Here is the proof:

http://img38.imageshack.us/img38/589/kinectsuccess.png

[onethought]

Beautiful!! Its really awesome to see the sample working on someone else's machine. :D

Do you have any suggestions on how I can improve the setup process / make it easier?
From the sounds of it I may need to add something that only starts the tracking when UDK is completely open.

[HackNFly]

Nope, the tracking worked beautifully once the Kinect Camera was tilted straight on and I was in frame. I have some C code that can be incorporated into your dll if you want that will let you control the motor. More importantly however would be some type of visual feedback of where you are in the frame before it locks in on you. You were speaking about this in the future improvements. Have you experimented with this / had any luck with it yet? I have some ideas, but I may hold off if you already have something in the works.

~Santiago

[onethought]

I'm starting on the Image Buffer, User Map, User Histogram and Depth Buffer feeds today. The way I would ideally like to get the information into unreal would be through getting direct access to the textures buffer and doing a memcpy over it. I currently have no idea on how to do this though.

I could always use the scripted texture class and draw each pixel, but that would be ludicrously slow to do in unreal script.

I think I'll post a question in the Programming / UnrealScript section of the forums but I am certainly open to suggestions.

[HackNFly]

Quote from Michael Allar:
"Using scripted textures I had to down sample to a 64x48 texture to get interactive frame rates in a very populated scene, it might be because i might not be using scripted textures correctly but iunno. I'm not going to share it simply because its just a very hacky very bad way of doing things."

My idea was to use the getUserPixels function to create an image similar to what is done in the players sample from NITE. Then make this a binary image and downsample. Then send the data through UDK to Scaleform where the pixels would be drawn on the screen programmatically.

I think the question has been posed before, and the current response is that without the UE3 license you can't have direct access to memcpy. Then again some people said you couldn't get the Kinect to work with UDK because of the same reasoning.

This will be a lot easier when Scaleform moves to AS3, then you can just create a "webcam feed" and play it directly. Or use the bitmap.draw function.

Also, since your alpha build has progressed to the point where multiple people have confirmed it works right, it might be a good time to move to some type of subversion. I would love to be able to check out the current work you're working on to test / fiddle with.

~Santiago

[onethought]

Ok, thanks for the heads up about the direct access thing. Although, I think I have an idea on how to do this but it's going to require a little bit of memory hacking. I'll find out if it works when I try it ;)

As for subversion, I will shift to it soon but I want to make sure features work before they are available for people to fiddle with. I know thats not a particularly great stance to have but I want a feature to be stable before I release it.

[HackNFly]

No, I completely understand. You're a person who clearly takes pride in their work. I will make use of the current api. I appreciate the work you have done.

[stevenchau432]

when I open the NIUI_Content.udk , immediately crash , error ... how can I do ?

I already to do some normal step :
1. installed the kinect drivers , it is ok , no problem , can UserTracker.
2. copy the Quick Start ->UDK-2010-12 cover the original file
3. run the frontend and full compile
4. open UDK Editor , view -> world info -> game type use NIUI_Main
5. open the NIUITestBed.udk and press play for here.

but ... kinect can't tracker any bone , and can't open NIUI_Content.udk .

onethought can you help me ? I don't know UnrealScript, sorry , my english is poor .

[HackNFly]

@onethought, you can jump in if any of what I say doesn't make sense.

@steven
From my experience you can't use dllbind from within the editor. You'll have to make a shortcut to the game.

here is a quote from @onethought


Alternatively, try creating a new shortcut and pasting the following as the shortcut location (don't browse to the UDK.exe, just paste this in):

"C:\UDK\UDK-2010-12\Binaries\Win32\UDK.exe" NIUITestBed?game=NIUI.NIUI_Main -vadebug -useunpublished -windowed -resx=1280 -resy=960 -log -ConsolePosX=0 -ConsolePosY=0


The new frontend made things some stuff a little harder to work with, you now have to use the ?game= when you run a game to specify the game type.

[stevenchau432]

@onethought, you can jump in if any of what I say doesn't make sense.

@steven
From my experience you can't use dllbind from within the editor. You'll have to make a shortcut to the game.

here is a quote from @onethought


Alternatively, try creating a new shortcut and pasting the following as the shortcut location (don't browse to the UDK.exe, just paste this in):

"C:\UDK\UDK-2010-12\Binaries\Win32\UDK.exe" NIUITestBed?game=NIUI.NIUI_Main -vadebug -useunpublished -windowed -resx=1280 -resy=960 -log -ConsolePosX=0 -ConsolePosY=0


The new frontend made things some stuff a little harder to work with, you now have to use the ?game= when you run a game to specify the game type.

Yes , yes , you right , THANK !
when I use the shortcut "C:\UDK\UDK-2010-12\Binaries\Win32\UDK.exe" NIUITestBed?game=NIUI.NIUI_Main -vadebug -useunpublished -windowed -resx=1280 -resy=960 -log -ConsolePosX=0 -ConsolePosY=0

I Success ! thank again.

[stevenchau432]

Beautiful!! Its really awesome to see the sample working on someone else's machine. :D

Do you have any suggestions on how I can improve the setup process / make it easier?
From the sounds of it I may need to add something that only starts the tracking when UDK is completely open.


I have some suggestions , next time , have you want to do the human to walk / jump / run / and Tracker the head rotation ?
now , NIUI only Tracker user 2d animation , can't walk / jump / run / enter vehicle...
but anyway , I very very thank onethought to do this work.

[HackNFly]

onethought has done some amazing things here, so I don't put it past him to be able to add some of those things you suggested, but just as some extra information, the head tracking currently isn't supported in the PrimeSense NITE algorithm. They simply lock your head in with your shoulders. You can tilt your head, but no rotation. Some other people have used other head trackers(i.e. OpenCV haar face tracker) and added that to the algorithm.

Just felt like adding some background information.

[onethought]

@stevenchau432

I'm going to work on a tutorial for combing the bone overrides with animations (IE: The character's legs are in a running animation but you can control the upper body). This won't be out for a little while though.
I also updated the quick-start guide and will be releasing it in the next version.

[stevenchau432]

@stevenchau432

I'm going to work on a tutorial for combing the bone overrides with animations (IE: The character's legs are in a running animation but you can control the upper body). This won't be out for a little while though.
I also updated the quick-start guide and will be releasing it in the next version.

WAOOOO! SO great for your concept , but I feel this work is very difficult . I don't Uscript , so I can't help you about program , sorry , but I support you .

[Aldgazar]

Hi,

I'm really loving this project. I've been trying to get my Kinect to work on Windows for a little while now and this is just what I was looking for. However, I am using Win64, so I was wondering if there was anything different I needed to do in the installation process (referencing the the Win32 folder in the instructions). Also, when you start this up, do you need to run a sensor program before running UDK or does it run on its own.

Thanks,
Ald

[HackNFly]

Just make sure you have the players.exe sample project from NITE working. Have to have all the drivers working first for it to work in UDk. Other than that you should be able to follow the quick install instructions and make it work.

[onethought]

@Aldgazar

Like HackNFly said, you should be able to setup the API by the way described in the Quick-Start Guide. That said, I haven't tried the API on 64bit Windows, so I am interested in hearing in how it goes.

Thanks for using my work! :)

[Aldgazar]

Got it to work. :) Not sure if there was anything special, but I did drop everything that was supposed to go into the Win32 folder into the Win64 folder as well.

[Aldgazar]

Does your API support the NITE events for waving and clicking (focus gesture)?
NITE has these in the documentation but I want to know if your API has the ability to recognize them.

Push Detector
Swipe Detector
Wave Detector

[onethought]

Does your API support the NITE events for waving and clicking (focus gesture)?
NITE has these in the documentation but I want to know if your API has the ability to recognize them.

Push Detector
Swipe Detector
Wave Detector

No it currently doesn't. I haven't used those features a whole lot yet, but I intend on putting them in when I get around to doing UI support. I am aiming for UI support to be in the final release (which I am aiming for Mid-July).

[narcissist]

I've got the NIUI sample working. Great work on this btw, I'm looking forward to seeing this project progress. It really opens up a lot of creative possibilities for 3D control systems in the UDK. However, before getting stuck into learning this interface for a project, I'd like to clarify something:

Does OpenNI/NITE provide functionality for tracking single points, or bones that aren't part of a human skeleton? (say you wanted to create a custom skeleton with more joints than usual, or track a separate object in the scene) OR, perhaps there are ways to do your own processing on the image+depth streams?

From what I've read, I'm thinking "no". I guess I'd have to use something like OpenCV? (not looking forward to that at all, after seeing the C++ work you did, and having no experience myself) In any case, I'll be playing around with your interface some more, it gave me a reason to buy a kinect, haha. Thanks.

[onethought]

@narcissist

Nice to hear you got the sample to work. :)

I don't believe that OpenNI / NITE provides the functionality to track single points. That said, you could do it yourself by extracting a user from the depth map with the label map and isolating points from that. You would need to do this in C++ though, as doing per-pixel operations in UScript is absurdly slow.

[Aldgazar]

Sorry to be so pushy, but how difficult would it be to implement those interactions I mentioned before? Unfortunately I can't wait until July.
I believe I understand a little bit of the C++ here. I would add the "XN_CALLBACK_TYPE Push_Pushed" to the library export file and add a pushDetected variable to the Core as well as an Event_pushDetected to the Event Manager.

[onethought]

Sorry to be so pushy, but how difficult would it be to implement those interactions I mentioned before? Unfortunately I can't wait until July.
I believe I understand a little bit of the C++ here. I would add the "XN_CALLBACK_TYPE Push_Pushed" to the library export file and add a pushDetected variable to the Core as well as an Event_pushDetected to the Event Manager.

It wouldn't be that hard to extend the API to use those features, you would need to create the Push, Swipe and Wave detector's in C++. You'd then need to link in the function pointers to those detectors so the events can be processed.

Lastly, you would also need to write the wrapper code (in C++ and UScript) to bring the data into UDK.

(Sorry about the late reply, I've been busy the last few days).

[onethought]

Hi all,

So I posted this last week but it got deleted, but I recently finished a video tutorial series on the API for Design3. You can find the series here (http://www.design3.com/udk/intermediate/niui-essentials).

Also, the beta version of the API is now released. This is still in early days and I haven't done any work on it for a little while now. If you find bugs, please let me know.

Features added:
- Camera feeds to external window (Depth, User and Image buffers).
- Initial Reference rotation caching to remove the need for T-Pose to be forced in game.

See updated first post for download link.

[HackNFly]

Wow, you got the camera working in game? How'd you accomplish this? I have a fast approaching deadline on our game, so I haven't had a chance to look through / install the newest beta, but I'm very interested in what approach you did for the preview window.

[onethought]

Its just an external window at the moment using a software renderer that I wrote a little while ago to display the images. I had tried using DX9 to send the images to the GPU and render but I ended up getting massive frame rate spikes in game. I think UDK and my dll were fighting for control of the graphics card hence the lags.
I still haven't gotten around to trying to get the images to stream onto textures in game yet.

[HackNFly]

I was having trouble getting it to work on my new computer using the quickstart guide. Just wanted to update you with the small issues I had.

Even though you mention it in the full guide, with the quickstart guide it didn't mention to move NIUITestBed.udk from ./UDKGame/Maps to ./UDKGame/Content/Maps. I still think thats the best place to put it, since the engine will search for it in that location if you do a default install of UDK-2010-12. And not everyone uses NFringe.

Also, neither guide mentions anything about adding

+ModEditPackages=NIUI

under


[UnrealEd.EditorEngine]
+EditPackages=UTGame
+EditPackages=UTGameContent
+ModEditPackages=NIUI

in DefaultEngine.ini in ./UDKGame/Config

I know this is one of the most basic steps required to make a game, so most people would know it, however it would be nice for completeness for someone never having used the engine before. Also its hard to trace this error if you don't know what to look for since the frontend will compile just fine without that step.

Also this might just be nitpicky, but figure I'll give you the feedback, once I got the demo to run, if I try running the restartlevel command, the program crashes. Does this happen on your end?

[HackNFly]

Now that the Official SDK is out, are you planning to update your code to be compliant with it?

[fxhuanghfx]

Hi, thank you for uploading so much useful information!
1. Would you please help me to solve some urgent problems?

I followed all your steps in video using these packages: UDKInstall-2010-12-BETA , NIUI API - Alpha v1.1, nFringeSetup-pre-1.2.14.400.msi.
Then NIUI configuration was finished, OPENNI and Kinect were initialized successfully . However, the psi person can not move to follow me.

Console Window showed the code:

NIUI Event NewUser was recieved for user 1.
NIUI: User 1 was detected.
NIUI Event PoseDetected was recieved for user 1.
NIUI Event CalibrationStart was recieved for user 1.
NIUI: User 1 Calibration has begun.
NIUI Event CalibrationFailed was recieved for user 1.
User 78912188 calibration has failed. NIUI Event PoseDetected was recieved for user 1.
NIUI Event CalibrationStart was recieved for user 1.
NIUI: User 1 Calibration has begun.
NIUI Event CalibrationFailed was recieved for user 1.
User 78912188 calibration has failed. NIUI Event PoseDetected was recieved for user 1.
NIUI Event CalibrationStart was recieved for user 1.
...

I was wondering whether the failure reason is that my computer is windows 7 64 bits.

2. I have downloaded the latest preview release nFringe software, but it can not be activated.Would you please tell me how can I activate nFringe.



Could you help me?
My email : fxhuanghfx@gmail.com


Thank you so much!!!

[fxhuanghfx]

Hi, thank you for uploading so much useful information!
1. Would you please help me to solve some urgent problems?

I followed all your steps in video using these packages: UDKInstall-2010-12-BETA , NIUI API - Alpha v1.1, nFringeSetup-pre-1.2.14.400.msi.
Then NIUI configuration was finished, OPENNI and Kinect were initialized successfully . However, the psi person can not move to follow me.

Console Window showed the code:

NIUI Event NewUser was recieved for user 1.
NIUI: User 1 was detected.
NIUI Event PoseDetected was recieved for user 1.
NIUI Event CalibrationStart was recieved for user 1.
NIUI: User 1 Calibration has begun.
NIUI Event CalibrationFailed was recieved for user 1.
User 78912188 calibration has failed. NIUI Event PoseDetected was recieved for user 1.
NIUI Event CalibrationStart was recieved for user 1.
NIUI: User 1 Calibration has begun.
NIUI Event CalibrationFailed was recieved for user 1.
User 78912188 calibration has failed. NIUI Event PoseDetected was recieved for user 1.
NIUI Event CalibrationStart was recieved for user 1.
...

I was wondering whether the failure reason is that my computer is windows 7 64 bits.

2. I have downloaded the latest preview release nFringe software, but it can not be activated.Would you please tell me how can I activate nFringe.



Could you help me?
My email : fxhuanghfx@gmail.com

Thank you so much!!!

[fxhuanghfx]

Hi! Can you help me?
I followed all the steps in Onethought's video using these packages: UDKInstall-2010-12-BETA , NIUI API - Alpha v1.1, nFringeSetup-pre-1.2.14.400.msi.
Then NIUI configuration was finished, OPENNI and Kinect were initialized successfully . However, the psi person can not move to follow me.

Console Window showed the code:
...
NIUI: User 1 was detected.
NIUI Event PoseDetected was recieved for user 1.
NIUI Event CalibrationStart was recieved for user 1.
NIUI: User 1 Calibration has begun.
NIUI Event CalibrationFailed was recieved for user 1.
User 78912188 calibration has failed. NIUI Event PoseDetected was recieved for user 1.
...
I was wondering whether the failure reason is that my computer is windows 7 64 bits.

I have downloaded the latest preview release nFringe software, but it can not be activated.Would you please tell me how can I activate nFringe.

My email: fxhuanghfx@gmail.com.
Thank you so much!!!

[onethought]

Hi guys,

Sorry for being off the radar for a little while but I've been doing a bit of contract work.

@HackNFly
Yes, I do plan on updating the API to support the Microsoft SDK. I have some other things I am doing at the moment but I intend on starting it within the next fortnight.

@fxhuanghfx
That doesn't sound like an issue with your computer being 64-bit, it wouldn't even be working to that extent if it was (its more likely UDK would crash on start-up). To me it sounds as though it can't successfully calibrate so make sure that the Kinect has a good view of you. The latest version of the API has an output window in the demo that you can use to figure out if the Kinect can see you correctly.
Also, make sure you aren't using the Kinect in an area that's in bright sunlight. I've noticed that it interferes with the IR camera.

[fxhuanghfx]

Hi, Onethought, thank for your suggestion yesterday. I succeeded !!!!!!!The skeleton can be tracked and the psi person can move to follow me .
After I adjusted the light

intensity in my room and put Kinect in the higher horizontal level, it worked now. Thank you so much!

But I still have some questions. Could you help me?

1.I found it is hard to ensure the success of Kinect work every time when I opened UDK.exe. I have tried to opened UDK.exe for almost ten times today, but only 4 times

attempts were successful. The rest trials failed and the console window frequently showed
“ user was detected.
Pose detected .
Calibration has begun.
Calibration failed

to received for user 1, Pose lost. ”

My kinect work was quite unstable. I do not know why. I did not change anything in these failure trials.
2. I found two problems in the running. Do you think it was caused by the following mistakes and warnings?


(1). I select DM-Deck , click Script and select full recompile in Unreal Frontend-C:\UDK\UDK-2012, then the window show the warning information below.

--------------------NIUI - Release--------------------
...
Warning/Error Summary
---------------------
C:\UDK\UDK-2010-12\Development\Src\NIUI\Classes\NIUI_Core.uc(242) : Warning, Function parameter: 'quat' conflicts with struct defined in 'Object'
Success - 0 error(s), 1 warning(s)
Execution of commandlet took: 33.96 seconds

[fxhuanghfx]

(2) I click the button in Unreal Frontend-C:\UDK\UDK-2012.
then the window show the warning
...
Warning/Error Summary
---------------------
Error, The script contained in package '..\..\UDKGame\Script\Core.u' was compiled in debug mode. Please recompile all script package in release before cooking.
Error, Failed loading ..\..\UDKGame\Script\Core.u
Error, The script contained in package '..\..\UDKGame\Script\Engine.u' was compiled in debug mode. Please recompile all script package in release before cooking.
Error, Failed loading ..\..\UDKGame\Script\Engine.u
Error, The script contained in package '..\..\UDKGame\Script\GFxUI.u' was compiled in debug mode. Please recompile all script package in release before cooking.
Error, Failed loading ..\..\UDKGame\Script\GFxUI.u
Error, The script contained in package '..\..\UDKGame\Script\GameFramework.u' was compiled in debug mode. Please recompile all script package in release before

cooking.
Error, Failed loading ..\..\UDKGame\Script\GameFramework.u
Error, The script contained in package '..\..\UDKGame\Script\UnrealEd.u' was compiled in debug mode. Please recompile all script package in release before cooking.
Error, Failed loading ..\..\UDKGame\Script\UnrealEd.u
Error, The script contained in package '..\..\UDKGame\Script\GFxUIEditor.u' was compiled in debug mode. Please recompile all script package in release before cooking.
Error, Failed loading ..\..\UDKGame\Script\GFxUIEditor.u
Error, The script contained in package '..\..\UDKGame\Script\IpDrv.u' was compiled in debug mode. Please recompile all script package in release before cooking.
Error, Failed loading ..\..\UDKGame\Script\IpDrv.u
Error, The script contained in package '..\..\UDKGame\Script\UDKBase.u' was compiled in debug mode. Please recompile all script package in release before cooking.
Error, Failed loading ..\..\UDKGame\Script\UDKBase.u
Error, The script contained in package '..\..\UDKGame\Script\UTGame.u' was compiled in debug mode. Please recompile all script package in release before cooking.
Error, Failed loading ..\..\UDKGame\Script\UTGame.u
Error, The script contained in package '..\..\UDKGame\Script\UTEditor.u' was compiled in debug mode. Please recompile all script package in release before cooking.
Error, Failed loading ..\..\UDKGame\Script\UTEditor.u
Error, The script contained in package '..\..\UDKGame\Script\OnlineSubsystemSteamworks.u' was compiled in debug mode. Please recompile all script package in

release before cooking.
Error, Failed loading ..\..\UDKGame\Script\OnlineSubsystemSteamworks.u
Error, The script contained in package '..\..\UDKGame\Script\NIUI.u' was compiled in debug mode. Please recompile all script package in release before cooking.
Error, Failed loading ..\..\UDKGame\Script\NIUI.u

Failure - 24 error(s), 0 warning(s)
Execution of commandlet took: 132.93 seconds

Thank you very much!

[MAB]

Hello,

I have problem with running NIUI.
I installed last versions of OpenNI and NITE.
Then i installed Kinect drivers from https://github.com/avin2/SensorKinect.

When i run UDK i got:
NIUI_Error: OpenNI Context failed to initilise.
Is the device OpenNI capable and is it plugged in?

I can successfully run OpenNI and NITE samples.

[Gordorodo]

Thank you very VERY much!!
I'm definetively using this.
I was wondernig if you know how i can import data to udk thorugh a UDP connection (i guess this is the way NIUI works).
I'm tracking with a webcam and i want the position of a trackeable object to be imported into UDK and asigned to for examlpe a cube.
Thanks again!
Sorry for bad english

[onethought]

Thank you very VERY much!!
I'm definetively using this.
I was wondernig if you know how i can import data to udk thorugh a UDP connection (i guess this is the way NIUI works).
I'm tracking with a webcam and i want the position of a trackeable object to be imported into UDK and asigned to for examlpe a cube.
Thanks again!
Sorry for bad english

NIUI works using DLLBind to directly link to OpenNI and then get the data generated by the Kinect into UDK.
As for the web-cam tracking, you could use TCPLink (http://udn.epicgames.com/Three/TcpLink.html)to do what you described with UDP (I personally haven't used it so I can't speak for this approach). The other way is to use DLLBind (http://udn.epicgames.com/Three/DLLBind.html)to directly link to your code. It can be a little tricky but the advantage is that you don't need to launch a third-party application to get your interactions into UDK.

[Gordorodo]

Thanks for the reply. I don't know why my last reply to this thread (made a couple of weeks ago) never showed up.
Do you know where i can find detailed information on how to do this with DLLBind (or TCPLink)?
I'm proccessing the camera feed in python using OpenCV and then sending the (x,y) postion of the tracked object through a TCP socket (this way i would use TCPLink)
What should i do for DLLBind.
I'm pretty new at this.
Thanks again!!!!

[chester1997]

..........

[JonathanLuc]

Hello everybody !

Firstly, thank you so much for NIUI! I've been really astonished by the results it has even with my 3-years old laptop!

I'm a french student working on a project aiming to manipulate 3D objects (generated with UDK) with the hand tracking of the kinect. That's why I've tried your NIUI API.

However, the computer I use for this project is a 64bits one, and I can't make NIUI work with it. To succeed in my laptop (32bits), I've had to recompile the NI Unreal Interface.sln and to use the new NIUI.dll it generates. But, I can't recompile it in a x64 release :(

I've tried to redirect the x86 files (in visual studio) to x64 files but I always have problem while opening udk.exe (execution stops before it fully opens).

Do you have the visual studio solution for 64bits device ? Or just the NIUI.dll generated with it ?
Or do you think the problem can come from anything else ?

regards

[fingolfin]

This may look like reviving an old thread, but I have some problem on initializing NIUI device. Log says NIUI has failed to initialize and then UDK crashes before the game is actually launched. Any suggestions?

It might have something to do with Frontend error when I full recompile scripts:

C:\UDK\UDK-2012-01\Development\Src\NIUI\Classes\NIUI_Core.uc(242) : Warning, Function parameter: 'quat' conflicts with struct defined in 'Object'

Edit: My mistake. Cleaning official Kinect drivers have solved the problem.

[ZenekZezool]

Hey!

I just wanted to share with you the results of our work. We have delevoped a game in which the user can actually that they are inside the computer world we created (based on movie "Tron: Legacy"). It wouldn't be possible without the NIUI API, so here I share my gratefulness :)

Here is the link to the movie:
https://www.youtube.com/watch?v=ch3TD6u461I


Cheers!

[karim]

hi onethought ,
thank you very much for sharing your work to others.
i had a small problem , i can not download NIUI API from mediafire, it required user name and password. what is the asked user name and password?
is it mediafire user name and password or something else ?